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Description 



SYSTEMS AND METHODS FOR INTEROPERABLE 
MULTIMEDIA CONTENT DESCRIPTIONS 



Background of the Invention 



I. 



Field of the invention. 



The present invention relates to techniques for describing multimedia 
information, and more specifically, to techniques which describe both video and 
image information as well as to content of such information. 

II. Description of the related art. 

With the maturation of the global Internet and the widespread employment 
of regional networks and local networks, digital multimedia information has 
become increasingly accessible to consumers and businesses. Accordingly, it has 
become progressively more important to develop systems that process, filter, 
search and organize digital multimedia information, so that useful information can 
be culled from this growing mass of raw information. 

At the time of filing the instant application, solutions exist that allow 
consumers and business to search for textual information. Indeed, numerous 
text-based search engines, such as those provided by yahoo.com, goto.com, 
excite.com and others are available on the World Wide Web, and are among the 
most visited Web sites, indicating the significant of the demand for such 
information retrieval technology. 
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Unfortunately, the same is not true for multimedia content, as no generally 
recognized description of this material exists. In this regard, there have been past 
attempts to provide multimedia databases which permit users to search for pictures 
using characteristics such as color, texture and shape information of video objects 

5 embedded in the picture. However, at the closing of the 20th Century, it is not yet 
possible to perform a general search the Internet or most regional or local networks 
for multimedia content, as no broadly recognized description of this material exists. 
Moreover, the need to search for multimedia content is not limited to databases, but 
extends to other applications, such as digital broadcast television and multimedia 

0 telephony. 

One industry wide attempt to develop such standard a multimedia 
description framework has been through the Motion Pictures Expert Group's 
("MPEG") MPEG-7 standardization effort. Launched in October 1996, MPEG-7 
aims to standardize content descriptions of multimedia data in order to facilitate 
5 content-focused applications like multimedia searching, filtering, browsing and 
summarization. A more complete description of the objectives of the MPEG-7 
standard are contained in the International Organisation for Standardisation 
document ISO/IEC JTC 1 /SC29/WG 1 1 N2460 (Oct. 1998), the content of which is 
incorporated by reference herein. 
0 The MPEG-7 standard has the objective of specifying a standard set of 

descriptors as well as structures (referred to as "description schemes") for the 
descriptors and their relationships to describe various types of multimedia 
information. MPEG-7 also proposes to standardize ways to define other 
descriptors as well as "description schemes" for the descriptors and their 
relationships. This description, i.e. the combination of descriptors and description 
schemes, shall be associated with the content itself, to allow fast and efficient 
searching and filtering for material of a user's interest. MPEG-7 also proposes to 
standardize a language to specify description schemes, i.e. a Description Definition 
Language ("DDL"), and the schemes for binary encoding the descriptions of 
multimedia content. 
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At the time of filing the instant application, MPEG is soliciting proposals 
for techniques which will optimally implement the necessary description schemes 
for future integration into the MPEG-7 standard. In order to provide such 
optimized description schemes, three different multimedia-application 
arrangements can be considered. These are the distributed processing scenario, the 
content- exchange scenario, and the format which permits the personalized viewing 
of multimedia content. 

Regarding distributed processing, a description scheme must provide the 
ability to interchange descriptions of multimedia material independently of any 
platform, any vendor, and any application, which will enable the distributed 
processing of multimedia content. The standardization of interoperable content 
descriptions will mean that data from a variety of sources can be plugged into a 
variety of distributed applications, such as multimedia processors, editors, retrieval 
systems, filtering agents, etc . Some of these applications may be provided by third 
parties, generating a sub-industry of providers of multimedia tools that can work 
with the standardized descriptions of the multimedia data. 

A user should be permitted to access various content providers* web sites to 
download content and associated indexing data, obtained by some low-level or 
high-level processing, and proceed to access several tool providers' web sites to 
download tools (e.g. Java applets) to manipulate the heterogeneous data 
descriptions in particular ways, according to the user's personal interests. An 
example of such a multimedia tool will be a video editor. A MPEG-7 compliant 
video editor will be able to manipulate and process video content from a variety of 
sources if the description associated with each video is MPEG-7 compliant. Each 
video may come with varying degrees of description detail, such as camera motion, 
scene cuts, annotations, and object segmentations. 

A second scenario that will greatly benefit from an interoperable content 
description standard is the exchange of multimedia content among heterogeneous 
multimedia databases. MPEG-7 aims to provide the means to express, exchange, 
translate, and reuse existing descriptions of multimedia material. 
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Currently, TV broadcasters, Radio broadcasters, and other content providers 
manage and store an enormous amount of multimedia material. This material is 
currently described manually using textual information and proprietary databases. 
Without an interoperable content description, content users need to invest 
manpower to translate manually the descriptions used by each broadcaster into 
their own proprietary scheme. Interchange of multimedia content descriptions 
would be possible if all the content providers embraced the same scheme and 
content description schemes. This is one of the objectives of MPEG-7. 

Finally, multimedia players and viewers that employ the description 
schemes must provide the users with innovative capabilities such as multiple views 
of the data configured by the user. The user should be able to change the display's 
configuration without requiring the data to be downloaded again in a different 
format from the content broadcaster. 

The foregoing examples only hint at the possible uses for richly structured 
data delivered in a standardized way based on MPEG-7. Unfortunately, no prior 
art techniques available at present are able to generically satisfy the distributed 
processing, content-exchange, or personalized viewing scenarios. Specifically, the 
prior art fails to provide a technique for capturing content embedded in multimedia 
information based on either generic characteristics or semantic relationships, or to 
provide a technique for organizing such content. Accordingly, there exists a need 
in the art for efficient content description schemes for generic multimedia 
information. 



Summary of the Invention 

An object of the present invention is to provide content description schemes 
for generic multimedia information. 

Another object of the present invention is to provide techniques for 
implementing standardized multimedia content description schemes. 



# 
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A further object of the present invention is to provide an apparatus which 
permits users to perform general searches on the Internet or regional or local 
networks for multimedia content. 

Still another object of the present invention is to provide a technique for 
capturing content embedded in multimedia information based on either generic 
characteristics or semantic relationships, 

Still a further object of the present invention is to provide a technique for 
organizing content embedded in multimedia information based on either generic 
characteristics or semantic relationships. 

In order to meet these and other objects which will become apparent with 
reference to further disclosure set forth below, the present invention provides a 
system for generating a description record from multimedia information. The 
system includes at least one multimedia information input interface receiving 
multimedia information, a computer processor, and a data storage system, 
operatively coupled to said processor, for storing said at least one description 
record. In order to satisfy the objectives of the present invention, the processor 
performs object extraction processing to generate multimedia object descriptions 
from the multimedia information, and object hierarchy processing to generate 
multimedia object hierarchy descriptions, to generate at least one description record - 
including the multimedia object descriptions and multimedia object hierarchy 
descriptions for content embedded within the multimedia information 

In a preferred arrangement, the multimedia information is image 
information, the multimedia object descriptions are image object descriptions, and 
the multimedia hierarchy object descriptions are image object hierarchy 
descriptions. In an alternative preferred arrangement, the multimedia information 
is video information, the multimedia object descriptions are video object 
descriptions, and the multimedia object hierarchy descriptions are video object 
hierarchy descriptions. 

Where the multimedia information is image information, it is highly 
preferred that the object extraction processing includes image segmentation 
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30 



processing to segment each image in the image information into regions, and 
feature extraction processing to generate one or more feature descriptions for one 
or more of the regions. The descriptions may include text annotations, color, 
texture, shape, size, and position information. 

Likewise, it is advantageous for the object hierarchy processing to include 
physical object hierarchy organization to generate physical object hierarchy 
descriptions of the image object descriptions that are based on spatial 
characteristics of the objects, and logical object hierarchy organization to generate 
logical object hierarchy descriptions of the image object descriptions that are based 
on semantic characteristics of the objects, such that the image object hierarchy 
descriptions comprise both physical and logical descriptions. An encoder may be 
added to the system for encoding the image object descriptions and image object 
hierarchy descriptions into compressed description information. 

Where the multimedia information is video information, it is highly 
1 5 preferred that the object extraction processing includes video segmentation 

processing to temporally segment the video information into one or more video 
events or groups of events, video object extraction processing to segment the video 
events into regions and generate feature descriptions for the regions; and feature 
extraction processing to generate one or more feature descriptions for the video 
events. The feature descriptions for events may include text annotations, shot 
transition, camera motion, time, and key frame. The feature descriptions for 
objects may include text annotations, color, texture, shape, size, position, motion, 
and time. 

Likewise, it is advantageous for the object hierarchy processing to include 
both physical event and object hierarchy organization to generate physical event 
and object hierarchy descriptions of the video event and object descriptions that are 
based on temporal characteristics of the video events and objects, and logical event 
and object hierarchy organization to generate logical event and object hierarchy 
descriptions of the video event and object descriptions that are based on semantic 
characteristics of said the objects, and video object hierarchy extraction processing 
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to generate hierarchy descriptions for events and objects embedded within the 
video information. 

The present invention also provides methods to provide a content 
description scheme for generic multimedia information. In one arrangement, the 
5 method includes the steps of receiving multimedia information, processing the 
multimedia information by performing object extraction processing to generate 
multimedia object descriptions; processing the generated multimedia object 
descriptions by object hierarchy processing to generate multimedia object hierarchy 
descriptions, so that at least one description record including the object descriptions 
0 and the hierarchy objects descriptions is generated for content embedded within the 
multimedia information; and storing the record. The multimedia information may 
be image or video information. 

The present invention further provides computer readable media containing 
digital information with at least one multimedia description record describing 
multimedia content for corresponding multimedia information. In one 
arrangement, the media includes at least one object description for corresponding 
objects embedded in the multimedia information, one or more features 
characterizing each of the objects; and any available hierarchy information relating 
at least a portion of the objects in accordance with at least one of the features. 
The multimedia information may be image or video information, and where video 
information, the objects may be events or video objects embedded within the video 
information. 

The accompanying drawings, which are incorporated and constitute part of 
this disclosure, illustrate a preferred embodiment of the invention and serve to 
explain the principles of the invention. 

Brief Description of the Drawings 

Fig. 1 is a system diagram of a preferred embodiment of the present 
invention; 



WO 00/28440 




PCT/US99/26125 



Fig. 2 is a functional diagram of a multimedia content description system 
suitable for employment in the system of Fig. 1 ; 

Fig. 3 is an illustrative diagram of an image showing exemplary image 

objects; 

Figs. 4a and 4b are illustrative diagrams showing a set of image objects and 
exemplary hierarchal organizations for the exemplary image objects shown in Fig. 
3; 

Fig. 5 is an illustrative diagram of an video showing exemplary video 

events; 

Figs. 6a and 6b are illustrative diagrams showing a set of video events and 
an exemplary hierarchal organization for the exemplary video objects shown in 
Fig. 5; 

Fig. 7 is a flow diagram of a process which may be implemented in the 
system of Fig. 1 to generate image descriptions; and 

Fig. 8 is a flow diagram of a process which may be implemented in the 
system of Fig. 1 to generate video descriptions. 

Description of the Preferred Embodiments 

Referring to Fig. 1, an exemplary embodiment of the present invention is 
provided. The architecture of the system 100 includes a client computer 1 10 and a 
server computer 120. The server computer 120 includes a display interface 130, a 
query dispatcher 140, a performance database 150, query translators 160, 161, 165, 
target search engines 170, 171, 175, an additional client computer 180, and 
multimedia content description systems 200, 201, 205, which will be described in 
further detail below. 

While the following disclosure will make reference to this exemplary 
client-server embodiment, those skilled in the art should understand that the 
particular system arrangement may be modified within the scope of the invention 
to include numerous well-known local or distributed architectures. For example, 
all functionality of the client-server system could be included within a single 
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computer, or a plurality of server computers could be utilized with shared or 
separated functionality. The multimedia content description systems 200, 201, 
205, are preferably software routines executing on a general purpose processor 
within the server computer 120. 

Commercially available metasearch engines act as gateways linking users 
automatically and transparently to multiple text-based search engines. The system 
of Fig. 1 grows upon the architecture of such metasearch engines and is designed to 
intelligently select and interface with multiple on-line mulimedia search engines by 
ranking their performance for different classes of user queries. Accordingly, the 
query dispatcher 140, query translators 160, 161, 165, and display interface 130 of 
commercially available metasearch engines may be employed in the present 



invention. 



The dispatcher 140 selects the target search engines to be queried by 
consulting the performance database 150 upon receiving a user query. This 
database 150 contains performance scores of past query successes and failures for 
each supported search option. The query dispatcher only selects search engines 
170, 171, 175 that are able to satisfy the user's query, e.g. a query seeking color 
information will trigger color enabled search engines. 

The query translators 160, 161, 165, translate the user query to suitable - 
scripts conforming to the interfaces of the selected search engines. The display 
component 130 uses the performance scores to merge the results from each search 
engine, and presents them to the user. 

In accordance with the present invention, in order to permit a user to 
intelligently search the Internet or a regional or local network for multimedia 
content, search queries may be made with respect to the content embedded in 
multimedia information. Content based search queries may be made by 
descriptions of multimedia content in accordance with description schemes of the 
present invention, by example or by sketch. Each search engine 1 70, 1 71, 1 75 
employs a description scheme, for example the description schemes described 
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below, to describe the contents of multimedia information accessible by the search 
engine and to implement the search. 

In order to implement a content-based search query for multimedia 
information generated via client computer 110, the dispatcher 140 will match the 
query description, through a the multimedia content description system 200, 
employed by each search engine 170, 171, 175 to ensure the satisfaction of the user 
preferences in the query. It will then select the target search engines 1 70, 1 71 , 1 75 
to be queried by consulting the performance database 1 50. If the user of client 
computer 1 10 wants to search by color and one search engine does not support any 
color descriptors, it will not be useful to query that particular search engine. 

Next, the query translators 160 will adapt the query description to 
descriptions conforming to each selected search engine. This translation will also 
be based on the description schemes available from each search engine. This task 
may require executing extraction code for standard descriptors or downloaded 
extraction code from specific search engines to transform descriptors. For example, 
if the user specifies the color feature of an object using a color coherence of 166 
bins, the query translator will translate it to the specific color descriptors used by 
each search engine, e.g. color coherence and color histogram of x bins. 

Before displaying the results to the user, the query interface will merge the 
results from each search option by translating all the result descriptions into a 
homogeneous one for comparison and raking. Again, similarity code for standard 
descriptors or downloaded similarity code from search engines may need to be 
executed. User preferences will determine how the results are displayed to the user. 

Alternatively, a search query can be entered via client computer 1 80 which 
directly interfaces with target search engine 1 70. Unlike a query entered through 
client computer 110, client computer 1 80 will not enable a metasearch via multiple 
search engines. However, the multimedia content description system 200 may be 
employed in either arrangement to conduct a content based search in accordance 
with the present invention. 
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Referring next to Fig. 2, a description system 200 which, in accordance 
with the present invention, is employed by each search engine 1 70. 171, 1 75 is now 
described. In the preferred embodiment disclosed herein, the Extensible Markup 
Language ("XML") is used to describe multimedia content. XML is a subset of 
the Standard Generalized Markup Language ("SGML"), the standard language for 
defining and using document formats. SGML allows documents to be 
self-describing, i.e. they describe their own grammar by specifying the tag set used 
in the document and the structural relationships that those tags represent. XML 
retains the key SGML advantages in a language that is designed to be vastly easier 
to learn, use, and implement than full SGML. A complete description of XML can 
be found at the World Wide Web Consortium's web page on XML. at 
http://www.w3.org/XML/, the contents of which is incorporated by reference 
herein. 

The description system 200 advantageously includes several image and 
video processing, analysis and annotation sub-systems 210, 220, 230, 240, 250, 
260, 270, 280 to generate a rich variety of descriptions for a collection of image 
and video items 205. Each subsystem is described in turn. 

The first subsystem 210 is a region-based indexing and searching system 
which extracts visual features such as color, texture, motion, shape, and size for 
automatically segmented regions of a video sequence. The system 210 
decomposes video into separate shots by scene change detection, which may be 
either abrupt or transitional (e.g. dissolve, fade in/out, wipe). For each shot, the 
system 210 estimates both global (i.e. the motion of dominant background), and 
camera motion, then segments, detects, and tracks regions across the frames in the 
shot computing different visual features for each region. For each shot, the 
description generated by this system is a set of regions with visual and motion 
features, and the camera motion. A complete description of the region-based 
indexing and searching system 210 is contained in co-pending PCT Application 
Serial No. PCT/US98/09 124, filed May 5, 1998, entitled tk An Algorithm and 
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System Architecture for Object-Oriented Content-Based Video Search/' the 
contents of which are incorporated by reference herein. 

As used herein, a "video clip" shall refer to a sequence of frames of video 
information having one or more video objects having identifiable attributes, such 
as, by way of example and not of limitation, a baseball player swinging a bat, a 
surfboard moving across the ocean, or a horse running across a prairie. A "video 
object" is a contiguous set of pixels that is homogeneous in one or more features of 
interest, e.g., texture, color, motion and shape. Thus, a video object is formed by 
one or more video regions which exhibit consistency in at least one feature. For 
example a shot of a person (the person is the "object" here) walking would be 
segmented into a collection of adjoining regions differing in criteria such as shape, 
color and texture, but all the regions may exhibit consistency in their motion 
attribute. 

The second subsystem 220 is an MPEG domain face detection system, 
which efficiently and automatically detects faces directly in the MPEG compressed 
domain. The human face is an important subject in video. It is ubiquitous in news, 
documentaries, movies, etc., providing key information to the viewer for the 
understanding of the video content. This system provides a set of regions with face 
labels. A complete description of the system 220 is contained in PCT Application 
Serial No. PCT/US 97/20024, filed November 4, 1997, entitled "A Highly Efficient 
System for Automatic Face Region Detection in MPEG Video," the contents of 
which are incorporated by reference herein. 

The third subsystem 230 is a video object segmentation system in which 
automatic segmentation is integrated with user input to track semantic objects in 
video sequences. For general video sources, the system allows users to define an 
approximate object boundary by using a tracing interface. Given the approximate 
object boundary, the system automatically refines the boundary and tracks the 
movement of the object in subsequent frames of the video. The system is robust 
enough to handle many real-world situations that are hard to model in existing 
approaches, including complex objects, fast and intermittent motion, complicated 
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backgrounds, multiple moving objects and partial occlusion. The description 
generated by this system is a set of semantic objects with the associated regions 
and features that can be manually annotated with text. A complete description of 
the system 230 is contained in U.S. Patent Application Serial No. 09/405,555, 
filed September 24, 1998, entitled "An Active System and Algorithm for Semantic 
Video Object Segmentation," the contents of which are incorporated by reference 
herein. 

The fourth subsystem 240 is a hierarchical video browsing system that 
parsers compressed MPEG video streams to extract shot boundaries, moving 
objects, object features, and camera motion, and. It also generates a hierarchical 
shot-based browsing interface for intuitive visualization and editing of videos. A 
complete description of the system 240 is contained in PCT Application Serial No. 
PCT/US 97/08266, filed May 16, 1997, entitled "A Method and Architecture for 
Indexing and Editing Compressed Video Over the World Wide Web," the contents 
of which are incorporated by reference herein. 

The fifth subsystem 250 is the entry of manual text annotations. It is often 
desirable to integrate visual features and textual features for scene classification. 
For images from on-line news sources, e.g. Clarinet, there is often textual 
information in the form of captions or articles associated with each image. This 
textural information can be included in the descriptions. 

The sixth subsystem 260 is a system for high-level semantic classification 
of images and video shots based on low-level visual features. The core of the 
system consists of various machine learning techniques such as rule induction, 
clustering and nearest neighbor classification. The system is being used to classify 
images and video scenes into high level semantic scene classes such as {nature 
landscape}, {city/suburb}, {indoor}, and {outdoor}. The system focuses on 
machine learning techniques because we have found that the fixed set of rules that 
might work well with one corpus may not work well with another corpus, even for 
the same set of semantic scene classes. Since the core of the system is based on 
machine learning techniques, the system can be adapted to achieve high 
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performance for different corpora by training the system with examples from each 
corpus. The description generated by this system is a set of text annotations to 
indicate the scene class for each image or each keyframe associated with the shots 
of a video sequence. A complete description of the system 260 is contained in S. 
Paek et al., "Integration of Visual and Text based Approaches for the Content 
Labeling and Classification of Photographs," ACM SIGIR'99 Workshop on 
Multimedia Indexing and Retrieval. Berkeley, C A (1999), the contents of which 
are incorporated by reference herein. 

The seventh subsystem 270 is model based image classification system. 
Many automatic image classification systems are based on a pre-defined set of 
classes in which class-specific algorithms are used to perform classification. The 
system 270 allows users to define their own classes and provide examples that are 
used to automatically learn visual models. The visual models are based on 
automatically segmented regions, their associated visual features, and their spatial 
relationships. For example, the user may build a visual model of a portrait in which 
one person wearing a blue suit is seated on a brown sofa, and a second person is 
standing to the right of the seated person. The system uses a combination of 
lazy-learning, decision trees and evolution programs during classification. The 
description generated by this system is a set of text annotations, i.e. the user 
defined classes, for each image. A complete description of the system 270 is 
contained in A. James et al.,"Model Based Classification of Visual Information for 
Content-Based Retrieval," Symp. Elec. Imaging: Multimedia Proc. and App. - 
Storage Retrieval and for Image and Video Databases VII, IS&T/SPIE 99 (1999), 
the contents of which are incorporated by reference herein. 

Other subsystems 280 may be added to the multimedia content description 
system 200, e.g., a subsystem used by collaborators to generate descriptions. 

In operation, the image and video content 205 may be a database of still 
images or moving video, a buffer receiving content from a browser interface 206, 
or a receptacle for live image or video transmission. The subsystems 210, 220, 
230, 240, 250, 260, 270, 280 operate on the image and video content 205 to 
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generate descriptions 211, 22 1 ? 231, 241, 251, 261, 271, 281 that include low level 
visual features of automatically segmented regions, user defined semantic objects, 
high level scene properties, classifications and associated textual information, as 
described above. Once all the descriptions for an image or video item are 
generated and integrated 290. the descriptions are input into a database 295, which 
the search engine 1 70 accesses. 

The process implemented by the subsystem 210, 220, 230, 240, 250, 260, 
270, 280 to generate descriptions 2 1 1 , 22 1 , 23 1 , 24 1 , 25 1 , 26 1 , 27 1 , 28 1 in a 
standard format are described below with reference to Figs. 7-8. It should be noted 
that certain of the subsystems, i.e., the region-based indexing and searching 
subsystem 210 and the video object segmentation system 230 may implement the 
entire description generation process, while the remaining subsystems implement 
only portions of the process and may be called on by the subsystems 210, 230 
during processing. In a similar manner, the subsystems 2 1 0 and 230 may be called 
on be each other for specific tasks in the process. 

The standard description schemes for images will now be described with 
reference to Figs. 3-4. Referring to Fig. 3, an exemplary image 300 with three 
persons is shown. The <object> element is the fundamental description element. 
Each <object> element has an identifier that is unique within a given image 
description. The identifier is expressed as an attribute of the <object> element e.g. 
<object id= ,, l ,, >. The <object> element also requires an attribute named type to 
distinguish between physical objects and logical objects. Physical objects usually 
correspond to continuous regions of the image with some descriptors in common 
(features, semantics, etc.) - in other words, real objects in the image. Logical 
objects are groupings of objects based on some high-level semantic relationships 
(e.g. faces or people). The <object> elements may also include two more 
attributes, object_ref and object_node_ref. The former allows deriving one object 
from an existing one, and the latter links back to nodes in the object hierarchy. The 
set of all objects identified in an image is included within the object set element 
(<object_set>). 
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Nine exemplary objects are shown in Fig. 3, including the entire family 

portrait 300, father 310, mother 320, child 330, parents 340, children 350. faces 

360, father's face 311, mother's face 32 1 . These objects may be expressed as a set 

of objects 0, 1, 2, 3, 4, 5, 6, 7, 8. as shown in Fig. 4a, with the entire family portrait 

300 being object 0, the father 3 1 0 being object 1 , the mother 320 being object 2. 

child 330 being object 3 } parents 340 being object 4, children 350 being object 5, 

faces 360 being object 6, the father's face 311 being object 7 and the mother's face 

321 being object 8. In this example, the identified objects are each physical objects 

with the exception of faces 360. which is a logical object. In XML, these image 

objects may be expressed as follows: 

<object_set> 

<!— Family portrait --> 

<object id="0" type= ,, PHYSICAL ,, > ... </object> 
<!- Father --> 

<object id="l " type="PHYSICAL"> ... </object> 
<!-- Mother --> 

<object id="2 M type="PHYSICAL"> ... </object> 
<!--Son --> 

<object id="3 M type= n PHYSICAL ,, > ... </object> 
<!-- Parents — > 

<object id="4" type="PHYSICAL M > ... </object> 
<!-- Children --> 

<object id= H 5" type="PHYSICAL M > ... </object> 
<!-- Faces --> 

<object id="6 n type="LOGICAL M > ... </object> 
<!-- Fathers face— > 

<object id="7" type="PHYSlCAL"> ... </object> 
<!-- Mother's face — > 

<object id="8" type="PHYSICAL"> ... </object> 
</object_set> 

In the XML description, each object element has a unique identifier within 
an image description. The identifier is expressed as an attribute of the object 
element (id). Another attribute of the object element (type) distinguishes between 
physical and logical objects. The content of each object element has been left 
empty to illustrate the overall structure of the image description. 
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The image description scheme is comprised of object elements that are 
combined hierarchically in one or more object hierarchy elements 
(<object_hierarchy>). The hierarchy is a way to organize the object elements in the 
object set element. Each object hierarchy consists of a tree of object node elements 
(<object_node>). Each object node points to an object. 

The objects in an image can be organized by their location in the image or 
by their semantic relationships. These two ways to group objects generate two 
types of hierarchies: physical and logical hierarchies. A physical hierarchy 
describes the physical location of the objects in the image. On the other hand, a 
logical hierarchy organizes the objects based on a higher level understanding of 
their semantics, similar to semantic clustering. 

Continuing with the image example of Fig. 3, two possible hierarchies are 
shown in Fig 4b, an object hierarchy which organizes objects physically 410 is 
shown, i.e., objects 4 and 5 are physically within object 0. A second object 
hierarchy which organizes objects logically 420 is shown, i.e., objects 7 and 8 are 
associated with object 6. In XML, these two hierarchies may be expressed as 
follows: 

<objectJiierarchy type= n PHYSICAL"> 

<!-- Family portrait --> 
<object_node id="9" object_ref="0"> 

<!-- Parents --> 

<object_node id="10 n object_ref="4 M > 
<object_node id=" 1 1 " object_ref=" 1 M > 
<object_node id="12 M object_ref="7"/> 
</object_node> 

<object_node id="I3" object_ref="2"> 
<object_node id="14" object_ref="8 H /> 
</object_node> 
</object_node> 



<!-- Children --> 

<object_node id="15" object_ref="5 M > 
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<object_node id="16" object_ref="3 , 7> 
</object_node> 

</object_node> 
</object_hierarchy> 

<object_hierarchy type="LOGICAL"> 
<!-- Faces --> 

<object_node id="17" object_ref="6"> 
<object_node id="18" object_ref="7*7> 
<object_node id="19" object_ref="8*7> 
</object_node> 

</object_node> 
</object_hierarchy> 

The type of hierarchy is included in the object hierarchy element as an 
attribute (type). The object node element has associated a unique identifier in the 
form of an attribute (id). The object node element references an object element by 
using the latter's unique identifier. The reference to the object element is included 
as an attribute (object_ref). An object element can include links back to nodes in 
the object hierarchy as an attribute. 

An object set element and one or more object hierarchy elements form the 
image element (<image>). The <object> elements in the <object_set> element are 
combined hierarchically in a <object_hierarchy> element. The attribute 
object_node_ref of <object> elements points to their corresponding nodes in the 
<object_hierarchy> elements. On the other hand, the attribute event_ref of 
<event_node> elements references back to the <object> element. 

An <object> element may contain an optional <location> element and the 
feature descriptor elements, such as <text_annotation>. <color>, <texture>, 
<shape>, <size>, <position>, <motion> and <time> elements. The <location> 
element contains a list of the physical locations of the image. The <time> and 
<motion> elements only have sense when the object belongs to a video sequence, 
to be described below. For example: 
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<!-- Father's face — > 

<object id= M 7> 

<location></location> 

<text_annotation></text_annotation> 

<color></color> 

<shape></shape> 

<position></position> 

</object> 



Appendix I presents the complete image description for the example image shown 
in Fig. 3. The image description scheme is summarized below in Table I. 





TABLE I | 


Element 


Must 
Contain 


May Contain 


Can be Contained in 


image 


objectset 


objecthierarchy(s) 


(root element) 


object_set 


object(s) 




image 


object 




location 

text_annotation 

color 

texture 

shape 

size 

position 

motion 

time 


object_set 


object_hierarchy 


object_node 




image 


object_node 




object_node(s) 


object_hierarchy 
objectnode(s) 



The location element contains pointers to the locations of the image. Note 
that annotations can be textual, visual or multimedia. These features can be 
extracted or assigned automatically, semi-automatically or manually. 
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When features are extracted automatically, the feature descriptors could 
include links to extraction and similarity matching code, and may even include 
annotation elements from external descriptions, as shown in the following 
example: 

<object id= M 4 M type="PHYSICAL" object_node_ref="12 16 M > <!-- Father's face 
— > 

<color> </color> 

<texture> 

<tamura> 

<tamura_value coarseness= H 0.0r' contrast="0.39" orientation="0.7'7> 

<code type="EXTRACTION" language= M JAVA" version="1.2"> 

<location> <location_site href="ftp://extraction.tamura.java , V> </location> 

</code> 

</tamura> 

</texture> 

<shape> </shape> 

<position> </position> 

<text_annotation xmlns:extAnDS= ,, http://www.other.ds/annotation.eIements H > 
<ext AnDS : Obj ect>Face</ext AnDS : Obj ect> 
</text_annotation> 
</object> 

A second example, contained in Appendix II hereto, illustrates the content 
of a particular image, which may include one or more distinct objects, in terms of 
the features of those objects including the location where the image is stored, text 
annotations, i.e., the name of the picture, the names of the persons in the picture, 
the location where the picture was taken, the event that is represented by the 
picture, and the date of the picture, color features in terms of LUV color, texture 
features in terms of Tamura texture, and size or dimensions of the image. Thus, 
information concerning the entire picture, e.g., the location where the image is 
stored, is descriptive of the object "id=oO," which represents the entire image. 
Other information concerns other objects within the image. For object " id=ol," is 
particularly described in the example in terms of text annotation (including the 
name of the person), color, texture, shape using eigenvalue analysis, size, and 
position in terms of a segmentation mask analysis. For object " id=o2, n only text 
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annotations are provided. Object " id=o3 M is a logical object corresponding to the 
concept of faces. 

The standard description schemes for video will now be described with 
reference to Figs. 5-6. Referring to Fig. 5, an exemplary video clip 500 with five 
temporal video events is shown. In the video description scheme, the <event> 
element is the fundamental description element. Each <event> element has an 
identifier that is unique within a given video DS description. The identifier is 
expressed as an attribute of the <event> element, e.g. <event id- T'>. 

The <event> element requires another attribute named type to distinguish 
different types of events. The attribute type can have three different values: shot, 
continuous_group_shots, and discontinuous_group_shots. Discontinuous group of 
shots will usually be associated together based on common features (e.g. 
background color) or high-level semantic relationships (e.g. actor on screen). The 
<event> elements may also include two more attributes, basic_event_ref and 
event_node_ref. The former allows deriving one event from an existing one, and 
the latter links back to nodes in the event hierarchy. The set of all events are found 
within the <event_set> element. 

Nine exemplary video events are shown in Fig. 5, including the entire video 
sequence 500, a scene where the tiger is stalking the prey 510, a scene where the 
tiger is chasing its prey 520, a scene where the tiger captures its prey 530, and a 
scene where the tiger is feeding 540. The later scene includes two events, one 
where the tiger is hiding the food 550, and the second where the tiger is feeding the 
young 560. These video events, which are parallel to image objects, may be 
expressed as a set of events 0, 1, 2, 3, 4, 5, 6, as shown in Fig. 6a, with the entire 
video sequence 500 being event 0, the scene where the tiger is stalking the prey 
510 being event 1, the scene where the tiger is chasing its prey 520 being event 2, 
the scene where the tiger captures its prey 530 being event 4, the scene where the 
tiger is feeding 540 being event 4, the scene where the tiger is hiding the food 550 
being event 5, and the scene where the tiger is feeding the young 560 being event 
6. In this example, the identified events are each shots, with the exception of event 
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4 ? which is of the type continuous group shots. Non temporally aligned events 

such as events 1 and 3 could be organized into discontinuous group shots. In XML, 

these image objects may be expressed as follows: 

<event_set> 
<!-- The Tiger — > 

<event id= M 0 H type="SHOT'> ... </event> 
<!-- Stalking the prey --> 
<event id="P type="SHOT"> ... </event> 
<!-- The chase — > 

<event id= w 2" type="SHOT"> ... </event> 
<!— The capture ~> 

<event id="3" type= M SHOT M > ... </event> 
<!-- Feeding — > 

<event id= n 4" type="CONTINUOUS_GROUP_SHOTS ,, > ... </event> 

<!— Feeding the young --> 

<event id= M 5" type="SHOT'> ... </event> 

<!-- Protecting the food --> 

<event id="6" type= H SHOT H > ... </event> 

</event_set> 

Note that each <event> element empty to clearly show the overall structure 
of the video description scheme. It is important to note that the selection and 
definition of an event in a given video is determined by the author of the 
description. The <event> element can correspond to either a shot or a scene of a 
video, or even a combination of these. 

The video description scheme is basically comprised of <event> elements 
that are combined hierarchically in a <event_hierarchy> element. The 
<event_hierarchy> element must contain a single <event_node> element. The 
<event_node> contains zero or more <event_node> elements and zero or more 
<objectJiierarchy> elements, as described above for the image description scheme. 
Each <event_node> element has associated a unique identifier. The identifier is 
expressed as an attribute of the elements e.g. <event_node id=T>. The hierarchy 
is a way to organize the <event> elements in the <event set> element. 
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The different events that form a video sequence may be grouped or 
organized in two different ways: by their location in the video sequence or by their 
semantic relationships. The <event_hierarchy> element includes an attribute, type, 
to distinguish between physical and logical hierarchies. A physical hierarchy will 
describe the temporal relationships of the events in the image. On the other hand, 
logical hierarchies will organize the events based on a higher level understanding 
of their semantics. 

Each <event_node> element consists of a reference to a <event> element, 
by using the unique identifiers associated with each <event>. The reference to the 
<event> element is given as an event_ref attribute. The video of Fig. 5 has the 
hierarchy shown in figure 6b. This hierarchy is expressed in XML as follows: 

<event_hierarchy type= M PHYSICAL"> 
<!-- The Tiger --> 

<event_node id="7" event__ref="0"> 
<!-- Stalking the prey --> 
<event_node id= M 8" event_ref='T7> 
<!— The chase — > 

<event_node id="9" event_ref="27> 
<!- Capture --> 

<event_node id= ,, 10 H event_ref="37> 
<!-- Feeding — > 

<event_node id= M 1 1 " event_ref= M 4"> 
<!— Hiding the food — > 
<event_node id="12" event_ref= ,, 5 , 7> 
<!-- Feeding the young— > 
<event_node id="13 H event_ref="67> 
<event_/node> 
</event_node> 
</event_hierarchy> 

An event set element and one or more even hierarchy elements form the 
video element (<video>). The video element symbolizes the video sequence being 
described. . The <event> elements in the <event_set> element are combined 
hierarchically in a <event_hierarchy> element. The attribute e vent_node_ref of 
<event> elements points to the corresponding nodes in the <event_hierarchy> 
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elements. On the other hand, the attribute event_ref of <event_node> elements 
references back to the <event> element. 

In the video description scheme, an <event> element can contain the 
following elements: 



<location> (optional) 
<transition> (optional) 
<text_annotation> (optional) 
<object_set> (optional) 
<camera_motion>(optional) 
<time>(optional) 
<key_frame>(zero or more) 



: <object_set> <text^annotation> and the <location> elements are defined above 
h respect to the image description scheme. The <transition> element describes 
transitions between shots. Thus, event 3 in the tiger video can be described as 



follows: 

<!-- Capture — > 
<event id="3"> 
<textannotation> 
<name_annotation> 

<concept>The capture of the prey</concept> 

</name_annotation> 

</text_annotation> 

<text_annotation> ... </text_annotation> 
<object_set> ... </object_set> 
<camera_motion> ... </camera_motion> 
<time> ... </time> 
<key_frame> ... </key_frame> 
</event> 



Appendix III presents the complete video description for the example i 



In the video description scheme, the event element contains features 
including location, shot transition (i.e. various within shot or across shot special 
effects), camera motion, time, key frame, annotation and object set elements, 
among others. The object element is defined in the image description scheme; it 



shown in Fig. 5. 
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represents the relevant objects in the event. As in the image description, these 
features can be extracted or assigned automatically or manually. For those features 
extracted automatically, the feature descriptors can include links to extraction and 
similarity matching code. For example, 

<event id="3 M type=*THYSICAL" event_node_ref^ M 10"> <!-- Capture --> 

<object_set> </object_set> 

<camera_motion> 

<backgroun_affine_model> 

<background_affine_motionjvalue> 

<panning direction= ,t NE , V> 

<zoom direction="IN M /> 

</background_affine_motion_val ue> 

<code type="DISTANCE M language="JAVA M version="1.0 M > 
<location> <location_site href= f, ftp://dist.bacground.affine n /> </location> 
</code> 

</background_affine_model> 
</camera_motion> 
<time> </time> 
</event> 

A second example contained in Appendix IV describes the content of a 
particular video sequence which may include one or more distinct events, in terms 
of the features of those events including the location where the video is stored, text 
annotations, i.e., the name of the video, the names of the persons in the video, the 
location where the video was taken, the event that is represented by the video, and 
the date of the video, the objects within that video sequence, the camera motion, 
the total time of the video sequence in terms of number of frames, and keyframes. 
This information concerning the entire video sequence is descriptive of the event 
id=E0. Other information concerns other objects within the image. 

The event hierarchy used to organize the described content is a physical 
hierarchy and describes temporal relationships. In this case, the only event is 
id=E0 which corresponds to the entire video. Within that event, two hierarchies are 
used to described the objects within the event, i.e., a physical and logical hierarchy, 
and parallel the physical and logical hierarchies described previously for the image 
example. 
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The process implemented by the system 200 to generate the image and 
video descriptions described with reference to Figs. 3-6 shall now be described 
with reference to Figs. 7-8. Fig. 7 is a flow diagram which illustrates a preferred 
process for generating descriptions for images. Digital image data 71 0 is applied 
to the computer system via link 7 1 1 . The image data may be non-compressed, or 
may be compressed in accordance with any suitable compression scheme, e.g., 
JPEG. 

The computer system, under the control of suitable application software, 
first performs object extraction 720 on the image data 710, in order to generate 
image objects. Object extraction 720 may take the form of a fully automatic 
processing operation, a semi-automatic processing operation, or a substantially 
manual operation in which objects are defined primarily through user interaction, 
such as via a user input device. 

In a preferred method, object extraction 720 consists of two subsidiary 
operations, namely image segmentation 725 and feature extraction and annotation 
726. For the image segmentation 725 step, any region tracking technique which 
partitions digital images into regions that share one or more common 
characteristics may be employed. Likewise, for the feature extraction and 
annotation step 326, any technique which generates features from segmented 
regions may be employed. The region-based indexing and searching subsystem 
210 described above is suitable for automated image segmentation and feature 
extraction; the video object segmentation system 230 described above is suitable 
example of a semi-automated image segmentation and feature extraction. Manual 
segmentation and feature extraction could alternatively be employed. 

The object extraction processing 720 generates an image object set 721 and 
optional related features such as annotations (collectively "image object 
descriptions"), which are preferably further processed by an object hierarchy 
extraction and construction module 730. Alternatively, the objects 721 could be 
directly stored in a database 740, or encoded by an XML encoder 750 or a binary 
encoder 760 and then stored 751, 752 in database 740. 
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The object hierarchy extraction and construction module 730 operates on 
the image object descriptions to generate image object hierarchy descriptions 73 1 . 
Preferably, both physical object hierarchy organization 735 and logical object 
hierarchy organization 736 are performed in parallel to generate descriptions 73 1 . 
The region-based indexing and searching subsystem 2 1 0 described above is 
suitable for automated image object hierarchy construction; the video object 
segmentation system 230 described above is suitable example of a semi-automated, 
object hierarchy construction Manual object hierarchy construction could 
alternatively be employed. 

The image object hierarchy descriptions 731 are either directly stored in a 
database 740, or are encoded by an XML encoder 750 or a binary encoder 760 and 
then stored 75 1 , 752 in database 740 as an image description record. Once the 
image description records have been stored in data base storage 740, they remain 
available in a useful format for access and use by other applications 770, such as 
search, filter and archiving applications for example, via bidirectional link 771. 

Fig. 8 is a flow diagram which illustrates a preferred process for generating 
descriptions for video. Digital video data 8 1 0 is applied to the computer system 
via link 811. The video data may be non-compressed, or may be compressed in 
accordance with any suitable compression scheme, e.g., MPEG-1, MPEG-2, 
MPEG-4, motion JPEG, H.261 or H.263. 

The computer system, under the control of suitable application software, 
first performs event and object extraction 820 on the video data 81 0, in order to 
temporally segment the video data 810 into video events and to locate video 
objects within the events. Video event and object extraction 820 may take the form 
of a fully automatic processing operation, a semi-automatic processing operation, 
or a substantially manual operation in which objects are defined primarily through 
user interaction, such as via a user input device. 

In a preferred method, the video event and object extraction process 820 
consists of three subsidiary operations, namely temporal video segmentation 825, 
object extraction 826 and feature extraction and annotation 827. For the 
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segmentation 825 step, the video is temporally partitioned into shots, continuous 
grous of shots, or discontinuous groups of shots, that share one or more common 
characteristics. For the object extraction 826 step, video objects are extracted from 
the video shots in a similar manner to the extraction of image objects from still 
images, except that motion and time information may be utilized. The feature 
extraction and annotation step 827 may be performed in parallel with the object 
extraction step 826 and operates on the temporally segmented video shots to 
generate features such as camera motion, key frames and text annotations. The 
region-based indexing and searching subsystem 210 described above is suitable for 
automated segmentation, object extraction and feature extraction; the video object 
segmentation system 230 described above is suitable example of a semi-automated 
segmentation, object extraction and feature extraction. Manual segmentation and 
extraction could alternatively be employed. 

The event and object extraction processing 820 generates an event and 
video object set 821 and optional related features such as annotations (collectively 
"video object descriptions"), which are preferably further processed by an event 
and object hierarchy extraction and construction module 830. Alternatively, the 
events and video objects 821 could be directly stored in a database 840, or encoded 
by an XML encoder 850 or a binary encoder 860 and then stored in database 840. 

The module 830 operates on the video object descriptions to generate video 
object hierarchy descriptions 831 . Preferably, the video object descriptions are 
operated on in parallel by both physical and logical operations. Thus, the video 
object descriptions may be subjected to both physical (temporal) event hierarchy 
organization 835 and logical event hierarchy organization 836 in parallel, and then 
to physical object hierarchy organization 837 and logical object hierarchy 
organization 838, so that both video events and objects embedded within those 
events are hierarchally organized. The region-based indexing and searching 
subsystem 210 described above is suitable for automated video object hierarchy 
construction; the video object segmentation system 230 described above is suitable 
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example of a semi-automated video object hierarchy construction. Manual video 
object hierarchy construction could alternatively be employed. 

The video object hierarchy descriptions 831 are either directly stored in a 
database 840 together with the video object descriptions, or encoded by an XML 
encoder 850 or a binary encoder 860 and then stored in database 840, as an video 
description record. Once the video description records have been stored in data 
base storage 840, they remain available in a useful format for access and use by 
other applications 870, such as search, filter and archiving applications for 
example, via bidirectional link 871. 

The foregoing merely illustrates the principles of the invention. Various 
modifications and alterations to the described embodiments will be apparent to 
those skilled in the art in view of applicants' teachings, herein. It will thus be 
appreciated that those skilled in the art will be able to devise numerous systems and 
methods which, although not explicitly shown or described herein, embody the $ 
principles of the invention and are thus within the spirit and scope of the invention. 'U 
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Appendix I 
image_ds.dtd: 
<!-- Image DS --> 

<!ELEMENT image (object_set ? object_hierarchy*)> 
<!ELEMENT object__set (object+)> 

<!ELEMENT object (location?, text_annotation?, color?, texture?, shape?, size 9 
position?, motion?, time?)> 
<!ATTLIST object 

id ID # REQUIRED 

object_ref IDREF #IMPLIED 

object_node_ref IDREFS #IMPLIED 

type (PHYSICAL|LOGICAL) #REQUIRED> 

<!-- External object location DTD — > 
<!ENTITY % location SYSTEM M location.dtd M > 
%location; 

<!-- External object annotation DTD — > 

<! ENTITY % text_annotation SYSTEM "tex^annotation.dtd'^ 
%text_annotation; 

<!-- External object color DTD — > 
<!ENTITY % color SYSTEM "color.dtd"> 
%color; 

<!-- External object texture DTD — > 
<!ENTITY % texture SYSTEM "texture.dtd"> 
%texture; 

<!-- External object shape DTD --> 

<! ENTITY % shape SYSTEM "shape.dtd"> 

%shape; 

<!-- External object size DTD «> 

<! ENTITY % size SYSTEM "size.dtd"> 

%size; 

<!-- External object position DTD -> 

<! ENTITY % position SYSTEM "position.dtd M > 
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%position; 

<!-- External object motion DTD — > 
<!ENTITY % motion SYSTEM "motion.dtd"> 
%motion; 

<!-- External object time DTD — > 
<!ENTITY % time SYSTEM M time.dtd M > 
%time; 



<!— Object hierarchy --> 

<!-- The attribute "type" is the hierarchy binding type --> 
<!ELEMENT object_hierarchy (object_node)> 
<! ATTLIST object_hierarchy 

type (LOGIC ALjSPATlAL) #REQUIRED> 

<!ELEMENT object_node (object_node*)> 
<! ATTLIST object_node 

id ID # REQUIRED 

object^ref IDREF #REQUIRED> 

<! ENTITY mpeg7 "ISO/IEC JTC 1 /SC29/WG 1 1 MPEG-7 M > 
<!-- Image DS end ~ > 



location.dtd: 

<!-- Description of resources 1 location~> 

<!- Objects, image, videos can be located/accessed at different locations --> 
<!ELEMENT location (location_site*)> 
<!ATTLIST location 

xml-link CDATA #FIXED "EXTENDED" 

role CDATA #IMPLIED 

title CDATA #IMPLIED 

show (EMBED|REPLACE|NEW) "EMBED" 

actuate (AUTO|USER) "USER" 

behavior CDATA #IMPLIED> 

<!— One location site — > 

<! ELEMENT location_site EMPTY> 

<!ATTLIST location site 
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xml-link CDATA #FIXED "LOCATOR" 

role CDATA #IMPLIED 

href CDATA # REQUIRED 

title CDATA #IMPLIED 

show (EMBED|REPLACE|NEW) "NEW" 

actuate (AUTO|USER) "USER" 

behavior CDATA #IMPLIED> 

<!ELEMENT code (location*)> 
<! ATTLIST code 

type (EXTRACTIONjDISTANCE) "EXTRACTION" 

language (C|JAVA|PERL) #REQUIRED 

version CDATA #REQUIRED> 

<!-- Description of resources 1 storage location --> 

text_annotation.dtd: 

<!— Text annotation features — > 

<! ELEMENT text_annotation (concept*, name_annotation? ? people_annotation?, 
location_annotation?, event_annotation?, date_annotation?, object_annotation?)> 

<!-- Name annotation --> 

<!ELEMENT name_annotation (concept*)> 

<!-- People annotation --> 

<!ELEMENT people_annotation (concept*)> 

<!-- Location annotation — > 

<!ELEMENT locationannotation (concept*)> 

<!-- Event annotation — > 

<! ELEMENT event_annotation (concept*)> 

<!— Date annotation --> 

<!ELEMENT date_annotation (concept*)> 

<!-- Object annotation — > 

<!ELEMENT object annotation (concept*)> 

<!— Concept — > 

<!ELEMENT concept (#PCDATA|code)*> 
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<!ATTLIST concept 

language CDATA "english" 

annotation (automatic|manual) "manual"> 

<!-- Text annotation features end — > 



color.dtd: 

<!-- Color features — > 

<! ELEMENT color (colorhist*, luv_color*)> 

<!— Color histogram feature — > 

<!ELEMENT color hist (color hist value, code*)> 

<!ATTLIST colorjiist 

length CDATA #REQUIRED 

color_space (RGB|OHTA|HSV|LUV) #REQUIRED 
quantization (uniform|non-uniform) #REQUIRED> 

<! ELEMENT color_hist_value (#PCDATA)> 
<!ATTLIST color_hist_value 

format CDATA #REQUIRED> 

<!-- LUV color feature — > 

<! ELEMENT luv_color (luv_color_value, code*)> 
<!ATTLIST luv_color 

length CDATA #REQUIRED> 

<! ELEMENT luvcolorvalue (luv_bin*)> 

<!ELEMENT luv_bin EMPTY> 
<!ATTLIST luv_bin 

1 CDATA #REQUIRED 

u CDATA ^REQUIRED 

v CDATA #RJEQUIRED> 

<!— Color features end --> 



texture.dtd: 

<!-- Texture features — > 
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<!ELEMENT texture (tamura?)> 

<!— Tamura texture feature — > 
<!ELEMENT tamura (tamura__value, code*)> 

<! ELEMENT tamura_value EMPTY> 

<!ATTLIST tamura_value 

coarseness CDATA #REQUIRED 
contrast CDATA # REQUIRED 
orientation CDATA #REQUIRED> 

<!-- Texture features end — > 



shape.dtd: 

<!— Shape features — > 

<!ELEMENT shape (eigenvalue_analysis*)> 
<!-- Eigenvalue analysis shape feature -> 

<!ELEMENT eigenvalue_analysis (eigenvalue_analysis_value, code*)> 
<!ATTLIST eigenvalue analysis 

length CDATA #REQUIRED> 

<! ELEMENT eigenvalue_analysis_value (eigenvalue *)> 

<! ELEMENT eigenvalue EMPTY> 
<! ATTLIST eigenvalue 

value CDATA #REQUIRED> 

<!— Shape features end --> 
size.dtd: 

<!-- Size features — > 
<!-- Size -> 

<!ELEMENT size (size_dimensions | size_num_pixels)> 

<!-- Dimensions (X,Y) > 
<!ELEMENT size_dimensions EMPTY> 
<!ATTLIST size_dimensions 
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x CDATA #REQUIRED 
y CDATA #REQUIRED> 

<!-- Number of pixels — > 

<! ELEMENT size_num_pixels EMPTY> 

<!ATTLIST size_num_pixels 

area CDATA #REQUIRED> 

<!-- Size features end — > 



position.dtd: 

<!-- Position features — > 

<! ELEMENT position (segmentation_mask_analysis*)> 
<!-- Segmentation mask position feature --> 

<!ELEMENT segmentation_mask_analysis (segmentation_mask_analysis_value, 
code*)> 

<!ELEMENT segmentationjnask_analysis_value (left-top_vertex, centroid)> 

<!ELEMENT left-top__vertex EMPTY> 
<!ATTLIST left-top_vertex 

x CDATA # REQUIRED 

y CDATA #REQUIRED> 

<! ELEMENT centroid EMPTY> 
<!ATTLIST centroid 

x CDATA #REQUIRED 

y CDATA #REQUIRED> 

<!-- Position features end — > 



motion.dtd: 

<!— Motion features — > 

<! ELEMENT motion (affine_model*)> 

<!-- Affine motion feature --> 

<!ELEMENT affine_modeI (affine_modeI_value, code*)> 
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<!ELEMENT affinemodelvalue (vector2d*)> 

<! ELEMENT vector2d EMPTY> 
<!ATTLIST vector2d 

x CDATA # REQUIRED 

y CDATA #REQUIRED> 

<!— Motion features end —> 



time.dtd: 

<!— Time duration features — > 

<!ELEMENT time (time_interval*|time_instant|time_span)> 

<!-- Continuos duration time (seconds or frames in video clip)-> 
<! ELEMENT time_interval EMPTY > 
<!ATTLIST time_interval 

unit (SECONDS|FRAMES) "SECONDS" 

start CDATA #REQUIRED 

end CDATA #REQUIRED> 

<!— Instant in time — > 

<! ELEMENT timeinstant EMPTY> 

<!ATTLIST time_instant 

unit (SECONDS(FRAMES) "SECONDS" 

instant CDATA #REQUIRED> 

<!— Continuos duration time — > 
<!ELEMENT time_span EMPTY > 
<!ATTLIST time_span 

unit (SECONDS|FRAMES) "SECONDS" 

span CDATA #REQUIRED> 

<!— Time duration features end — > 



Farnily_Portrait.xml: 

<?xml version=" 1 .0" standalone="no"?> 

<!DOCTYPE image PUBLIC "ISO//mpeg7//xml//dtd//image_ds" 
"http://www.ee.columbia.edu/mpeg7/xml/dtd/image_ds.dtd"> 
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<image> 

<object_set> 

<!-- Family portrait --> 

<object id="0 H type="PHYSICAL M object_node_ref=''9"> 
<location> 

<location_site href="http://wwu'.family.portrait.gif7> 

</location> 

<text_annotation> 

<narneannotation> 

<concept>Family Portrait</concept> 

</name_annotation> 

<date_annotation> 

<concept>September 26th, 1 998</concept> 

</date_annotation> 

</text_annotation> 

<color> 

<Iuv_color length- T'> 
<luv_color_value> 

<Iuv_bin 1="56.70" u="4.67" v="78.567> 

</luv_co!or_value> 

</luv_color> 

</color> 

</object> 

<!-- Father »> 

<object id= M l H type— "PHYSICAL" object_node_ref="l 1"> 

<text_annotation> 

<people_annotation> 

<concept>Father</concept> 

</people_annotation> 

</text annotation> 

</object> 

<!-- Mother — > 

<object id="2" type="PHYSICAL" object_node_ref^"13"> 

<text_annotation> 

<people_annotation> 

<concept>Mother</concept> 

</people_annotation> 

</text_annotation> 



</object> 
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<f-Son -> 

<object id="3" type- M PHYSICAL n object_node_ref= M 16 M > 

<text_annotation> 

<people_annotation> 



</people_annotation> 
</text_annotation> 

</object> 

<!-- Parents — > 

<object id= M 4 n type^'THYSICAL" object_node_ref= M 10"> 

<text_annotation> 

<people_annotation> 

<concept>Parents</concept> 

</people_annotation> 

</text_annotation> 

</object> 

<!-- Children --> 

<object id="5" type= M PHYSICAL M object_node_ref="15 ,, > 

<text_annotation> 

<peopl e_annotation> 

<concept>Children</concept> 

</people_annotation> 

</text_armotation> 

</object> 

<!-- Faces — > 

<object id="6" type="LOGICAL" object_node_ref= M 17 M > 

<text_annotation> 

<people_annotation> 

<concept>Faces</concept> 

</people_annotation> 

</textannotation> 

</object> 

<!— Father's face— > 

<object id="7" type="PHYSICAL" object_node_ref= M 12 18"> 

<text_annotation> 

<people_annotation> 



<concept>Son</concept> 
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<concept>Father's face</concept> 

</people_annotation> 

</text_annotation> 

</object> 

<!-- Mother's face — > 

<object id-'^" type= M PHYSICAL n object_node_ref="14 19"> 

<text_annotation> 

<people_annotation> 

<concept>Mother's face</concept> 

</people_annotation> 

</text_annotation> 

</object> 
</object_set> 

<object_hierarchy type= M PHYSICAL M > 
<!— Family portrait > 
<object_node id= ,, 9" object_ref= ,, 0 ,, > 
<!-- Parents — > 

<object_node id=" 1 0 M object_ref= H 4 ,, > 
<object_node id=" 11" object_ref="l M > 
<object_node id="12" object_ref="77> 
</obj ect_node> 

<object_node id= f, 13" object_ref="2"> 

<object__node id="14" object_ref= n 8'7> 

</object_node> 

</object_node> 

<!-- Children --> 

<object_node id="l 5" object_ref= M 5"> 
<object_node id="16" object_ref= M 3"/> 
</objectnode> 
</obj ect_node> 
</object_hierarchy> 

<object_hierarchy type="LOGICAL M > 
<!— Faces — > 

<object_node id="l 7" object_re£="6 f, > 
<object_node id^'MS" object_ref="77> 
<object_node id="19" object_ref= M 87> 
</object_node> 
</ obj ect_node> 
</object_hierarchy> 
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</image> 
Appendix II 

?xml version= M 1 .0" standalone="no"?> 

<!DOCTYPE image PUBLIC M ISO//mpeg7//xml//dtd//image_ds M "image_ds.dtd"> 

<image> 

<object_set> 



<object id="o0" type= M PHYSICAL l, > 
<location> 

<location site 

hjef= n http://www.eexolumbia.edu/-syp/images/yosemite.gifV> 

</location> 



<text_annotation> 

<name_annotation> 

<concept> Yosemite's Nevada 

Falls</concept> 

<concept language= M spanish M >Cataratas del 

Nevada en Yosemite</concept> 

</name_annotation> 

<people_annotation> 

<concept>Seungyup Paek</concept> 
<concept>Alex Jaimes</concept> 

</people_annotation> 

<] ocat i on annotation> 

<concept>Yosemite , s Nevada 

Falls</concept> 

<concept 

annotation= n automatic">outdoor</concept> 

<concept 

annotation="automatic">landscape</concept> 

</location_annotation> 
<event_annotation> 

<concept>Trip to Nevada Falls in 

Yosemite</concept> 

</event_annotation> 
<date_annotation> 

<concept>September 26th, 1998</concept> 
</date_annotation> 
</text annotation> 
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<color> 

<luv_color length="l "> 

<luv_coIor_value> 

<luv bin 1="56.70" u="4.67" 

v='78.567> 

</luv_color_value> 
</luv_color> 
</color> 

<texture> 

<tamura> 

<tamura_value coarseness= ,, 0.70 M 

contrast="0.67 M orientation= ,, 0.22 ,, /> 

</tamura> 
</texture> 

<size> 

<size_dimensions x= M 512" y="7347> 

</size> 
</object> 

<object id= n ol M type =,, PHYSICAL"> 
<text_annotation> 

<name_annotation> 

<concept>Seungyup Paek</concept> 
</name_annotation> 
<peopl e_annotation> 

<concept>Seungyup Paek</concept> 
</people _annotation> 
</text_annotation> 

<color> 

<luv coIor length=" 1 M > 

<luv_color_value> 

<Iuv_bin l= n 56.70 M u="4.67 n 

</luv_color_value> 
</Iuv_color> 
</color> 

<texture> 

<lamura> 

<tamura_value coarseness= M 0.70 n 



v="78.567> 



contrast= f, 0.67 M orientation= H 0.227> 
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</tamura> 
</texture> 

<shape> 

<eigenvalue_analysis length="2"> 

<eigenvalue_analysisjvalue> 

<eigenvalue value-' 1 .22 7> 
<eigenvalue value="0.227> 
</eigenva!ue_analysis_value> 
</e i gen val ueanaly sis> 
</shape> 

<size> 

<size_num_pixels area="7347> 

</size> 

<position> 

<segmentation_mask_anaIysis> 

<segmentation_mask_analysis_value> 

<left-top_vertex x="23" y="45"A> 
<centroid x="35 M y="577> 
</segmentation_mask_anaIysis_value> 
</segmentation_mask_analysis> 
</position> 
</object> 

<object id= H o2" type="PHYSICAL"> 

<text_annotation> 

<concept>Seungyup's face</concept> 

</text_annotation> 

<!-- Other tags --> 
</object> 

<object id="o3" type="LOGICAL"> 
<text_annotation> 

<concept>Faces</concept> 
</text_annotation> 
</object> 
</object_set> 

<object_hierarchy type= "PHYSIC AL"> 

<object_node id= M o4" object_ref="oO ,, > 

<object_node id="o5 M object_ref="or> 

<object_node id="o6 M object_ref="o2 ,, /> 
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</object_node> 
</object_node> 
</obj ect_hierarchy> 

<object_hierarchy type= H LOGICAL"> 

<object_node id="o7 M object_ref= ,, o3 ,, > 

<object_node id="o8 M object_ref= M o27> 
</object_node> 
</object_hierarchy> 
</image> 
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Appendix HI 
video_ds.dtd: 
<!-- Video DS --> 

<!ELEMENT video (event_set, event_hierarchy*)> 
<! ELEMENT event_set (event+)> 

<! ELEMENT event (location?, transition?, text_annotation?, object_set?, 
camera_motion?, time?, key_frame*)> 
<!ATTLIST event 

id ID #REQUIRED 

event_ref IDREF ^IMPLIED 

event node ref IDREFS #IMPLIED 

type 

(SHOT|CONTINUOUSj3ROUP_SHOTS|DISCONTINUOUS GROUP SHOTS 
#REQUIRED> 

<!— External image DS DTD — > 

<! ENTITY % imagers SYSTEM " imagers. dtd M > 

%image_ds; 

<!-- Example of transitions: 

- dissolve (additive, cross, non-additive), 

- slide (band, slash, normal, band, boxes), 

- wipe (radial, random, rectangular, moving, crossed, star-shaped, corner, 
soft, cross-shaped, band, soft, gradient, diamond-shaped, pointed, circular, shaped), 

- merge (center), 

- peel (center, page, back), 

- stretch (cross, image, in, over), 

- spin (cube, rectangular, image, away), 

- zoom, curtain, door, funnel, spiral boxes, paint splatter, motion, 
luminance, push, flip, fold up, etc. 



<!ELEMENT transition EMPTY> 
<! ATTLIST transition 

effect CD ATA #REQUIRED> 

<!-- External camera motion descriptor DTD — > 

<! ENTITY % camera_motion SYSTEM "camera_motion.dtd M > 

%camera_motion; 



— > 
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<! ELEMENT key frame (size_dimensions?, time_instant?)> 



<!-- Event hierarchy — > 

<!-- The attribute "type M is the hierarchy binding type — > 
<!ELEMENT eventhierarchy (event_node)> 
<!ATTLIST event_hierarchy 

type (LOGICAL)SPATIAL) #REQUIRED> 

<!ELEMENT event_node (event_node* 5 object_hierarchy*)> 
<! ATTLIST event_node 

id ID # REQUIRED 

eventref IDREF #REQUIRED> 

<!-- Video DS end --> 



camera_motion.dtd: 

<!-- Camera motion features --> 

<! ELEMENT camera_motion (background_affine_motion*)> 
<!— Affine model for camera motion detection — > 

<!ELEMENT background_affine_motion (background_affine_motion_yalue, 
code*)> 

<!ELEMENT background_affine_motion_value (panning?, zoom?)> 

<! ELEMENT panning EMPTY> 
<! ATTLIST panning 

direction (NT|NE|ET|SE|ST|SW|WT|NW) #REQUIRED> 

<! ELEMENT zoom EMPTY> 
<! ATTLIST zoom 

direction (IN|OUT) #REQUIRED> 

<!-- Camera motion features end --> 
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<?xml version=" 1 .0 n standalone= M no"?> 

<!DOCTYPE video PUBLIC n ISO//mpeg7//xml//dtd//video_ds" "video_ds.dtd n > 
<video> 

<event_set> 

<event id="e0" type-"CONTINUOUS_GROUP_SHOTS ,, > 
<location> 

<location_site href="yosemite.avi/ M /> 
</location> 



<text_annotation> 

<name_annotation> 

<concept>Yosemite's Nevada 

Falls</concept> 

<concept language= M spanish">Cataratas del 

Nevada en Yosemite</concept> 

</name_annotati on> 
<people_annotation> 

<concept>Seungyup Paek</concept> 
</people_annotation> 
<Iocation_annotation> 

<concept>Yosemite's Nevada 

Falls</concept> 

<concept 

annotation="automatic n >outdoor</concept> 

<concept 

annotation= ,, automatic">landscape</concept> 

</location_annotation> 
<event _annotation> 

<concept>Trip to Nevada Falls</concept> 
</event_annotation> 
<date_annotation> 

<concept>September 26th, 1998</concept> 
</date_annotation> 
</text_annotation> 

<object_set> 

<object id="oO M type= ,, PHYSICAL ,, > 
<text_annotation> 

<name annotation> 
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Paek</concept> 



Paek</concept> 



u= H 4.67 H v= M 78.56'7> 



<concept>Seungyup 

</name_annotation> 
<people_annotation> 

<concept>Seungyup 

</people_annotation> 
</text_annotation> 
<color> 

<luv_color length=" 1 M > 

<luv_color_value> 

<luv bin I="56.70" 



</luv_color_value> 
</luv_color> 
</color> 
<texture> 

<tamura> 

<tamura value 
coarseness="0.70" contrast="0.67" orientation= M 0.22'7> 

</tamura> 
</texture> 
<shape> 

<eigenvalue_analysis length="2"> 

<eigenvalue_analysis_value> 
<eigenvalue 



value= M 1.22'7> 
value="0.22'7> 



<eigenvalue 

</eigenvalue_analysis value> 
</eigenvaIue_analysis> 
</shape> 
<size> 

<size_num_pixels area="734"/> 

</size> 
<position> 

<segmentation_mask_analysis> 



<segmentation_mask_analysis_vaIue> 

x="23 M y= M 45"/> 

y= ,, 57 , 7> 



<left-top_vertex 
<centroid x="35 M 
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</segmentation_mask_analysis_value> 

</segmentation_mask_analysis > 
</position> 
<motion> 

<affine_model> 

<affine_model_value> 

<vector2d x="12.3" 



y="2.34'7> 
y="12.347> 
y= ,, 23.34 ,, /> 



start= M l" end="3'7> 



<vector2d x="1.3 M 

<vector2d x="0.3" 

</affine_model_value> 
</affine_model> 
</motion> 
<time> 

<time_interval unit="FRAMES" 

</time> 
</object> 

<object id="o] n type="PHYSICAL M > 

<text_annotation> 

<concept>Seungyup's face</concept> 

</text_annotation> 

<!-- Other tags — > 
</object> 

<object id="o2" type= M LOGICAL"> 
<text_annotation> 

<concept>Faces</concept> 
</text_annotation> 
</object> 
</object_set> 

<carnera_motion> 

<background_afflne_motion> 

<background_affine_motion_value> 
<panning direction="SE"/> 
<zoom direction- 'IN"/^ 
</background_affine_motion_value> 
</background_affine_motion> 
</camera motion> 



<time> 
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<time_interval unit="FRAMES" start="l M 

end="107> 

</time> 



<key_frame> 

<size_dimensions x="512" y= M 734 n /> 
<time_instant unit="FRAMES" instant= M 5 M /> 

</key_frame> 



</event> 
</event_set> 



<event_hierarchy type^'THYSICAL'^ 

<event_node id="el M event_ref= n eO H > 

<object_hierarchy type="SPATIAL n > 

<object__node id="o3" object_ref="oO"> 

<object_node id="o4" object_ref= n or7> 
</object_node> 
</obj ect_hi erarchy > 
<object_hierarchy type="LOGICAL"> 

<object_node id="o5 M object_ref="o3"> 

<object_node id="o6" object_ref="ol7> 
</object_node> 
</object_hierarchy> 
</event_node> 
</event_hierarchy> 



</video> 



WO 00/28440 so w PCT/US99/26I25 



Claims 

We claim: 

1 1 . A system for generating a description record from multimedia information, 

2 comprising: 

3 (a) at least one multimedia information input interface receiving said 

4 multimedia information; 

5 (b) a computer processor, coupled to said at least one multimedia 

6 information input interface, receiving said multimedia information 

7 therefrom, processing said multimedia information by performing 

8 object extraction processing to generate multimedia object 

9 descriptions from said multimedia information, and processing said 
generated multimedia object descriptions by object hierarchy 
processing to generate multimedia object hierarchy descriptions, 
wherein at least one description record including said multimedia 
object descriptions and said multimedia object hierarchy 

14 descriptions is generated for content embedded within said 

*~ multimedia information; and 

16 < c ) a data storage system, operatively coupled to said processor, for 

1 7 storing said at least one description record. 

1 2. The system of claim 1 , wherein said multimedia information comprises 

2 image information, said multimedia object descriptions comprise image object 

3 descriptions, and said multimedia object hierarchy descriptions comprise image 

4 object hierarchy descriptions. 



10 
11 
12 
13 



1 

2 
3 



3. 



The system of claim 2, wherein said object extraction processing comprises: 
(a) image segmentation processing to segment each image in said 
image information into regions within said image; and 



WO 00/28440 9 su 9 PCT/US99/26I25 



(b) feature extraction processing to generate one or more feature 
descriptions for one or more of said regions; 
whereby said generated object descriptions comprise said one or more feature 
descriptions for one or more of said regions. 

4. The system of claim 3 , wherein, said one or more feature descriptions are 
selected from the group consisting of text annotations, color, texture, shape, size, 
and position. 

5. The system of claim 2, wherein said object hierarchy processing comprises 
physical object hierarchy organization to generate physical object hierarchy 
descriptions of said image object descriptions that are based on spatial 
characteristics of said objects, such that said image object hierarchy descriptions 
comprise physical descriptions. 

6. The system of claim 5, wherein, said object hierarchy processing further 
comprises logical object hierarchy organization to generate logical object hierarchy 
descriptions of said image object descriptions that are based on semantic 
characteristics of said objects, such that said image object hierarchy descriptions 
comprise both physical and logical descriptions. 

7. The system of claim 6 ? wherein said object extraction processing comprises: 

(a) image segmentation processing to segment each image in said 
image information into regions within said image; and 

(b) feature extraction processing to generate object descriptions for one 
or more of said region; 

and wherein said physical hierarchy organization and said logical hierarchy 
organization, generate hierarchy descriptions of said object descriptions for said 
one or more of said regions. 
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8. The system of claim 7, further comprising an encoder receiving said image 
object hierarchy descriptions and said image object descriptions, and encoding said 
image object hierarchy descriptions and said image object descriptions into 
encoded description information, wherein said data storage system is operative to 
store said encoded description information as said at least one description record. 

9. The system of claim 1 , wherein said multimedia information comprises 
video information, said multimedia object descriptions comprise video object 
descriptions including both event descriptions and object descriptions, and said 
multimedia hierarchy descriptions comprise video object hierarchy descriptions 
including both event hierarchy descriptions and object hierarchy descriptions. 

10. The system of claim 9, wherein said object extraction processing comprises: 

(a) temporal video segmentation processing to temporally segment said 
video information into one or more video events or groups of video 
events and generate event descriptions for said video events, 

(b) video object extraction processing to segment said one or more 
video events or groups of video events into one or more regions, 
and to generate object descriptions for said regions; and 

(c) feature extraction processing to generate one or more event feature 
descriptions for said one or more video events or groups of video 
events, and one or more object feature descriptions for said one or 
more regions; 

wherein said generated video object descriptions include said event feature 
descriptions and said object descriptions. 

1 1 . The system of claim 1 0, wherein said one or more event feature 
descriptions are selected from the group consisting of text annotations, shot 
transition, camera motion, time and key frame, and wherein said one or more 
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object feature descriptions are selected from the group consisting of color, texture, 
shape, size, position, motion, and time. 

12. The system of claim 9, wherein said object hierarchy processing comprises 
physical event hierarchy organization to generate physical event hierarchy 
descriptions of said video object descriptions that are based on temporal 
characteristics of said video objects, such that said video hierarchy descriptions 
comprise temporal descriptions. 

13. The system of claim 12, wherein said object hierarchy processing further 
comprises logical event hierarchy organization to generate logical event hierarchy 
descriptions of said video object descriptions that are based on semantic 
characteristics of said video objects, such that said hierarchy descriptions comprise 
both temporal and logical descriptions. 

14. The system of claim 13, wherein said object hierarchy processing further 
comprises physical and logical object hierarchy extraction processing, receiving 
said temporal and logical descriptions and generating object hierarchy descriptions 
for video objects embedded within said video information, such that said video 
hierarchy descriptions comprise temporal and logical event and object descriptions. 

15. The system of claim 14, wherein said object extraction processing 
comprises: 

(a) temporal video segmentation processing to temporally segment said 
video information into one or more video events or groups of video 
events and generate event descriptions for said video events, 

(b) video object extraction processing to segment said one or more 
video events or groups of video events into one or more regions, 
and to generate object descriptions for said regions; and 
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(c) feature extraction processing to generate one or more event feature 
descriptions for said one or more video events or groups of video 
events, and one or more object feature descriptions for said one or 
more regions; 

wherein said generated video object descriptions include said event feature 
descriptions and said object descriptions, and wherein said physical event hierarchy 
organization and said logical event hierarchy organization generate hierarchy 
descriptions from said event feature descriptions, and wherein said physical object 
hierarchy organization and said logical object hierarchy organization generate 
hierarchy descriptions from said object feature descriptions 

1 6. The system of claim 15, further comprising an encoder receiving said video 
object hierarchy descriptions and said video object descriptions, and encoding said 
said video object hierarchy descriptions and said video object descriptions into 
encoded description information, wherein said data storage system is operative to 
store said encoded description information as said at least one description record. 

17. A method for generating a description record from multimedia information, 
comprising the steps of: 

(a) receiving said multimedia information; 

(b) processing said multimedia information by performing object 
extraction processing to generate multimedia object descriptions 
from said multimedia information; 

(c) processing said generated multimedia object descriptions by object 
hierarchy processing to generate multimedia object hierarchy 
descriptions, wherein at least one description record including said 
multimedia object descriptions and said multimedia object hierarchy 
descriptions is generated for content embedded within said 
multimedia information; and 

(d) storing said at least one description record. 
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1 8. The method of claim 1 7, wherein said multimedia information comprises 
image information, said multimedia object descriptions comprise image object 
descriptions, and said multimedia object hierarchy descriptions comprise image 
object hierarchy descriptions. 

19. The method of claim 2, wherein said object extraction processing step 
comprises the sub-steps of: 

(a) image segmentation processing to segment each image in said 
image information into regions within said image; and 

(b) feature extraction processing to generate one or more feature 
descriptions for one or more of said regions; 

whereby said generated image object descriptions comprise said one or more 
feature descriptions for one or more of said regions. 

20. The method of claim 1 9, wherein, said one or more feature descriptions are 
selected from the group consisting of text annotations, color, texture, shape, size, 
and position. 

21 . The method of claim 1 8, wherein said step of object hierarchy processing 
includes the sub-step of physical object hierarchy organization to generate physical 
object hierarchy descriptions of said image object descriptions that are based on 
spatial characteristics of said objects, such that said image hierarchy descriptions 
comprise physical descriptions. 

22. The method of claim 21 , said step of object hierarchy processing further 
includes the sub-step of logical object hierarchy organization to generate logical 
object hierarchy descriptions of said image object descriptions that are based on 
semantic characteristics of said objects, such that said image object hierarchy 
descriptions comprise both physical and logical descriptions. 
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23. The method of claim 22, wherein said step of object extraction processing 
further includes the sub-steps of: 

(a) image segmentation processing to segment each image in said 
image information into regions within said image; and 

(b) feature extraction processing to generate object descriptions for one 
or more of said region; 

and wherein said physical object hierarchy organization sub-step and said logical 
object hierarchy organization sub-step generate hierarchy descriptions of said 
object descriptions for said one or more of said regions. 

24. The method of claim 24, further comprising the step of encoding said 
image object descriptions and said image object hierarchy descriptions into 
encoded description information prior to said data storage step. 

25. The method of claim 1 7, wherein said multimedia information comprises 
video information, said multimedia object descriptions comprise video object 
descriptions including both event descriptions and object descriptions, and said 
multimedia hierarchy descriptions comprise video object hierarchy descriptions 
including both event hierarchy descriptions and object hierarchy descriptions. 

26. The method of claim 25, wherein said step of object extraction processing 
comprises the sub-steps of: 

(a) temporal video segmentation processing to temporally segment said 
video information into one or more video events or groups of video 
events and generate event descriptions for said video events, 

(b) video object extraction processing to segment said one or more 
video events or groups of video events into one or more regions, 
and to generate object descriptions for said regions; and 
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9 (c) feature extraction processing to generate one or more event feature 

1 0 descriptions for said one or more video events or groups of video 

1 1 events, and one or more object feature descriptions for said one or 

12 more regions; 

13 wherein said generated video object descriptions include said event feature 

14 descriptions and said object descriptions. 



1 27. The method of claim 26, wherein said one or more event feature 

2 descriptions are selected from the group consisting of text annotations, shot 

3 transition, camera motion, time and key frame, and wherein said one or more 

4 object feature descriptions are selected from the group consisting of color, texture, 

5 shape, size, position, motion, and time. 

6 28. The method of claim 25, wherein, said step of object hierarchy processing 

7 includes the sub-step of physical event hierarchy organization to generate physical 

8 event hierarchy descriptions of said video object descriptions that are based on 

9 temporal characteristics of said video objects, such that said video hierarchy 
1 0 descriptions comprise temporal descriptions. 

1 29. The method of claim 28, wherein said step of object hierarchy processing 

2 further includes the sub-step of logical event hierarchy organization to generate 

3 logical event hierarchy descriptions of said video object descriptions that are based 

4 on semantic characteristics of said video objects, such that said hierarchy 

5 descriptions comprise both temporal and logical descriptions. 

1 30. The method of claim 29, wherein said step of object hierarchy processing 

2 further comprises the sub-step physical and logical object hierarchy extraction 

3 processing, receiving said temporal and logical descriptions and generating object 

4 hierarchy descriptions for video objects embedded within said video information, 
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such that said video hierarchy descriptions comprise temporal and logical event and 
object descriptions.. 

3 1 . The method of claim 30 ? wherein said step of object extraction processing 
comprises the sub-steps of: 

(a) temporal video segmentation processing to temporally segment said 
video information into one or more video events or groups of video 
events and generate event descriptions for said video events, 

(b) video object extraction processing to segment said one or more 
video events or groups of video events into one or more regions, 
and to generate object descriptions for said regions; and 

(c) feature extraction processing to generate one or more event feature 
descriptions for said one or more video events or groups of video 
events, and one or more object feature descriptions for said one or 
more regions; 

wherein said generated video object descriptions include said event feature 
descriptions and said object descriptions, and wherein said physical event hierarchy 
organization and said logical event hierarchy organization generate hierarchy 
descriptions from said event feature descriptions, and wherein said physical object 
hierarchy organization and said logical object hierarchy organization generate 
hierarchy descriptions from said object feature descriptions. 

32. The method of claim 1 5, further comprising the step of encoding said video 
object descriptions and said video object hierarchy descriptions into encoded 
description information prior to said data storage step. 



33. A computer readable media containing digital information with at least one 
multimedia description record describing multimedia content for corresponding 
multimedia information, the description record comprising: 
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(a) one or more multimedia object descriptions describing 
corresponding multimedia objects; 

(b) one or more features characterizing each of said multimedia object 
descriptions; and 

(c) one or more multimedia object hierarchy descriptions, if any, 
relating at least a portion of said one or more multimedia objects in 
accordance with one or more characteristics. 

34. The computer readable media of claim 33, wherein said multimedia 
information comprises image information, said multimedia objects comprise image 
objects, said multimedia object descriptions comprise image object descriptions, 
and said multimedia object hierarchy descriptions comprise image object hierarchy 
descriptions. 

35. The computer readable media of claim 34, wherein, said one or more 
features are selected from the group consisting of text annotations, color, texture, 
shape, size, and position. 

36. The computer readable media of claim 34, wherein said image object 
hierarchy descriptions comprise physical object hierarchy descriptions of said 
image object descriptions based on spatial characteristics of said image objects. 

37. The computer readable media of claim 36, wherein, said image object 
hierarchy descriptions further comprises logical object hierarchy descriptions of 
said image object descriptions based on semantic characteristics of said image 
objects. 

38. The computer readable media of claim 33, wherein said multimedia 
information comprises video information, said multimedia objects comprise events 
and video objects, said multimedia object descriptions comprise video object 
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descriptions including both event descriptions and object descriptions, said features 
comprise video event features and video object features, and said multimedia 
hierarchy descriptions comprise video object hierarchy descriptions including both 
event hierarchy descriptions and object hierarchy descriptions. 

39. The computer readable media of claim 38, wherein, said one or more event 
feature descriptions are selected from the group consisting of text annotations, shot 
transition, camera motion, time and key frame, and wherein said one or more 
object feature descriptions are selected from the group consisting of color, texture, 
shape, size, position, motion, and time.. 

40. The computer readable media of claim 38, wherein said event hierarchy 
descriptions comprise one or more physical hierarchy descriptions of said events 
based on temporal characteristics. 

41 . The computer readable media of claim 40, wherein said event hierarchy 
descriptions further comprise one or more logical hierarchy descriptions, of said 
events based on semantic characteristics. 

42. The computer readable media of claim 38, wherein said object hierarchy 
descriptions comprise one or more physical hierarchy descriptions of said objects 
based on temporal characteristics. 



43. The computer readable media of claim 39, wherein said object hierarchy 
descriptions further comprise one or more logical hierarchy descriptions, of said 
objects based on semantic characteristics. 
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AMENDED CLAIMS 

[received by the International Bureau on 12 May 2000 (12.05.00); 
original claims 1, 17 and 33 amended; remaining claims unchanged (3 pages)] 



6 
7 
8 
9 



1 1 . A system for generating a description record from multimedia information, 

2 comprising: 

3 (a) at least one multimedia information input interface receiving said 

4 multimedia information; 

5 (b) a computer processor, coupled to said at least one multimedia 
information input interface, receiving said multimedia information 
therefrom, processing said multimedia information by performing 
object extraction processing to generate multimedia object 
descriptions from said multimedia information, and processing said 

1 0 generated multimedia object descriptions by object hierarchy 

1 1 processing to generate multimedia object hierarchy descriptions 
indicative of an organization of said object descriptions , wherein at 
least one description record including said multimedia object 
descriptions and said multimedia object hierarchy descriptions is 

15 generated for content embedded within said multimedia 

16 information; and 

1 7 < c > a da 1 * storage system, operatively coupled to said processor, for 

1 8 storing said at least one description record. 

1 2. The system of claim 1 , wherein said multimedia information comprises 

2 image information, said multimedia object descriptions comprise image object 

3 descriptions, and said multimedia object hierarchy descriptions comprise image 

4 object hierarchy descriptions. 



12 
13 
14 



9 



1 3. The system of claim 2, wherein said object extraction processing comprises: 

(a) image segmentation processing to segment each image in said 
3 image information into regions within said image; and 

AMENDED SHEET (ARTICLE 19) 
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9 (c) feature extraction processing to generate one or more event feature 

1 0 descriptions for said one or more video events or groups of video 

1 1 events, and one or more object feature descriptions for said one or 

12 more regions; 

13 wherein said generated video object descriptions include said evem feature 
descriptions and said object descriptions, and wherein said physical event hierarchy 
organization and said logical event hierarchy organization generate hierarchy 
descriptions from said event feature descriptions, and wherein said physical object 
hierarchy organization and said logical object hierarchy organization generate 

1 8 hierarchy descriptions from said object feature descriptions 

1 16. The system of claim 15, further comprising an encoder receiving said video 

2 object hierarchy descriptions and said video object descriptions, and encoding said 

3 said video object hierarchy descriptions and said video object descriptions into 

4 encoded description information, wherein said data storage system is operative to 

5 store said encoded description information as said at least one description record. 

1 17. A method for generating a description record from multimedia information, 

2 comprising the steps of: 

3 (a) receiving said multimedia information; 

* (b) processing said multimedia information by performing object 

5 extraction processing to generate multimedia object descriptions 

6 from said multimedia information; 

7 (c) Proving said generated multimedia object descriptions by object 
hierarchy processing to generate multimedia object hierarchy 
descriptions indicative of an organization of said object 
descriptions, wherein at least one description record including said 
multimedia object descriptions and said multimedia object hierarchy 
descriptions is generated for content embedded within said 

1 3 multimedia information; and 

1 4 ( d ) storing said at least one description record. 

AMENDED SHEET (ARTICLE 19) 
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4 (a) one or more multimedia object descriptions describing 

5 corresponding multimedia objects; 

6 (b) one or more features characterizing each of said multimedia object 

7 descriptions; and 

8 (c) one or more multimedia object hierarchy descriptions indicative of 

9 an organization of said object descriptions, if any, relating at least a 

1 0 portion of said one or more multimedia objects in accordance with 

1 1 one or more characteristics. 



1 34. The computer readable media of claim 33, wherein said multimedia 

2 information comprises image information, said multimedia objects comprise image 

3 objects, said multimedia object descriptions comprise image object descriptions, 

4 and said multimedia object hierarchy descriptions comprise image object hierarchy 

5 descriptions. 



1 35. The computer readable media of claim 34. wherein, said one or more 

2 features are selected from the group consisting of text annotations, color, texture, 

3 shape, size, and position. 

1 36. The computer readable media of claim 34 s wherein said image object 

2 hierarchy descriptions comprise physical object hierarchy descriptions of said 

3 image object descriptions based on spatial characteristics of said image objects. 

1 37. The computer readable media of claim 36, wherein, said image object 

2 hierarchy descriptions tunher comprises logical object hierarchy descriptions of 

3 said image object descriptions based on semantic characteristics of said image 

4 objects. 



38. The computer readable media of claim 33, wherein said multimedia 
information comprises video information, said multimedia objects comprise events 
and video objects, said multimedia object descriptions comprise video object 
AMENDED SHEET (ARTICLE 19) 
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name of country.) 

CHANG, SHIH-FU 

560 Riverside Drive, Apt. 18K 

New York, NY 10027 

US 



State (that is, country) of nationality: 
TW 



State (that is, country) of residence: 
US 



Name and address: (Family name followed by given name; for a legal entity, full official designation. The address must include postal code and 
name of country.) 



State (that is, country) of nationality: 



State (that is, country) of residence: 



Name and address: (Family name followed by given name; for a legal entity, full official designation. The address must include postal code and 
name of country.) 



State (that is, country) of nationality: 



State (that is, country) of residence: 



Name and address: (Family name followed by given name; for a legal entity, full official designation. The address must include postal code and 
name of country.) 



State (that is, country) of nationality: 



State (that is, country) of residence: 



j | Further applicants are indicated on another continuation sheet. 
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Sheet No. 



ational application No. 
PCT/US99/26125 



Box No. Ill AGENT OR COMMON REPRESENTATIVE; OR ADDRESS FOR CORRESPONDENCE 



The following person is PQ agent I I common representative 

and has been appointed earlier and represents the applicants) also for international preliminary examination. 

is hereby appointed and any earlier appointment of (an) agent(s) /common representative is hereby revoked. 

| I is hereby appointed, specifically for the procedure before the International Preliminary Examining Authority, in 
' — * addition to the agent(s)/common representative appointed earlier. 



Name and address: (Family name followed by given name; for a legal entity, full official designation. 
The address must include postal code and name of country.) 

TANG, HENRY and 

RAG USA, PAUL A. 

Baker Botts LLP 

30 Rockefeller Plaza 

New York, NY 10112-0228 

US 



Telephone No.: 
(212) 705-5000 



Facsimile No.: 
(212) 705-5020 



Teleprinter No.: 



□ Address for correspondence: Mark this check-box where no agent or common representative is/has been appointed and 
the space above is used instead to indicate a special address to which correspondence should be sent. 



Box No. IV BASIS FOR INTERNATIONAL PRELIMINARY EXAMINATION 



Statement concerning amendments:* 

1 . The applicant wishes the international preliminary examination to start on the basis of: 

the international application as originally filed 
the description DO as originally filed 

| | as amended under Article 34 

the claims | | as originally filed 

PQ as amended under Article 19 (together with any accompanying statement) 
[ | as amended under Article 34 

the drawings OG as originally filed 

I | as amended under Article 34 

2. I I The applicant wishes any amendment to the claims under Article 19 to be considered as reversed. 

3. | | The applicant wishes the start of the international preliminary examination to be postponed until the expiration of 

20 months from the priority date unless the International Preliminary Examing Authority receives a copy of any 
amendments made under Article 19 or a notice from the applicant that he does not wish to make such amendments 
(Rule 69. 1 (d)). (This check-box may be marked only -where the time limit under Article 19 has not yet expired.) 
* Where no check-box is marked, international preliminary examination will start on the basis of the international application as 
originally filed or, where a copy of amendments to the claims under Article 19 and/or amendments of the international 
application under Article 34 are received by the International Preliminary Examining Authority before it has begun to draw up 
a written opinion or the international preliminary examination report, as so amended. 



Language for the purposes of international preliminary examination: English 

which is the language in which the international application was filed. 

□ 

which is the language of a translation furnished for the purposes of international search. 
□ which is the language of publication of the international application. 

□ 

which is the language of the translation (to be) furnished for the purposes of international preliminary examination. 



Box No. V ELECTION OF STATES 



The applicant hereby elects all eligible States (that is, all States which have been designated and which are bound by Chapter 11 of the 
PCT) 

excluding the following States which the applicant wishes not to elect: 



Form PCT/IPEA/401 (second sheet) (July 1998; reprint January 2000) 
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'nternational application No. 
PCT/US99/26125 



Box No. VI CHECKLIST 



The demand is accompanied by the following elements, in the language referred to in 
Box No. IV, for the purposes of international preliminary examination: 



1 ■ translation of international application 

2. amendments under Article 34 

3. copy (or, where required, translation) 
of amendments under Article 19 



copy (or, where required, translation) 
of statement under Article 19 



5. letter 

6. other (specify) 



sheets 
sheets 

4 sheets 

sheets 
sheets 
sheets 



For International Preliminary 
Examining Authority use only 

received not received 



□ 
□ 

□ 

□ 
□ 
□ 



□ 
□ 

□ 

□ 
□ 
□ 



The demand is also accompanied by the item(s) marked below: 

1 . fee calculation sheet 

2. | [ separate signed power of attorney 

3 I I copy of general power of attorney; 
I ' reference number, if any: 



4. | | statement explaining lack of signature 

5. I I nucleotide and or amino acid sequence 
' — ' computer readable form 

6. other (specify): Transmittal Letter 



Box No. VII SIGNATURE OF APPLICANT, AGENT OR COMMON REPRESENTATIVE 



Next to each signature, indicate the name of the person signing and the capacity in which the person signs (if such capacity is not 
obvious from reading the demand). 



/ Paul- 



Paul A. Ragusa (Agent) 



For International Preliminary Examining Authority use only 
1 . Date of actual receipt of DEMAND: 



2. Adjusted date of receipt of demand due 
to CORRECTIONS under Rule 60. 1(b): 



3. I I The date of receipt of the demand is AFTER the expiration of 19 months I I The applicant has been 

1 — ' from the priority date and item 4 or 5, below, does not apply. ' — ' informed accordingly. 



4. I I The date of receipt of the demand is WITHIN the period of 19 months from the priority date as extended by virtue of 
I — I Rule 80.5. 



5. I I Although the date of receipt of the demand is after the expiration of 19 months from the priority date, the delay in arrival is 
I ' EXCUSED pursuant to Rule 82. 



— ^— — ^— — ^ P or i memal j ona | B ureau use on iy — — _____ ^ _ 
Demand received from IPEA on: 

Form PCT/IPEA/40 1 (last sheet) (July 1 998; reprint January 2000) Legaistar 2000. Form pctdem See Notes (Q (ne demand f orm 
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CHAPTER II 



FEE CALCULATION SHEET 
Annex to the Demand for international preliminary examination 

■ For International Preliminary Examining Authority use only 



International 
application No. 


PCT/US99/26125 


Applicant's or agent's 
file reference 


32095-PCT 



Date stamp of the IPEA 



Applicant 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK 



Calculation of prescribed fees 



1 . Preliminary examination fee 

2. Handling fee (Applicants from certain States are 
entitled to a reduction of 75% of the handling fee. 
Where the applicant is (or all applicants are) so 
entitled, the amount to be entered at H is 25% of the 
handling fee.) 

3. Total of prescribed fees 

Add the amounts entered at P and H 

and enter total in the TOTAL box 



490.00 



153.00 H 



643.00 



TOTAL 



authorization to charge deposit 
account with the IPEA (see below) 



Mode of Payment 

□ 

|X| cheque 

| | postal money order 

| | bank draft 



| I cash 

I I revenue stamps 

| I coupons 

| | other (specify): 



Deposit Account Authorization (this mode of payment may not be available at all IPEAs) 

The IPEA/ US | | is hereby authorized to charge the total fees indicated above to my deposit account. 



02-4377 



(this check-box may be marked only if the conditions for deposit accounts of the IPEA so permit) is 
hereby authorized to charge any deficiency or credit any overpayment in the total fees indicated 
above to my deposit account. 



17 May 2000 



Deposit Account Number 



Date (day/month/year) 




enature 



Form PCT/IPEA/401 (Annex) (July 1 998; reprint January 2000) Legaistar 2000, Form pctdfee 



See Notes to the fee calculation sheet 
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r^ENT COOPERATION TREA 



PCT/US99/26125 

pc4 



From the INTERNATIONAL BUREAU 



PCT 

NOTIFICATION CONCERNING 

CI IDIUIICCinKI OR TRAMQMITTAI 

OF PRIORITY DOCUMENT 

(PCT Administrative Instructions, Section 41 1) 


BAKER B0TTS LLP. 

TANG, Henry 00JAN2 ' * * IS 

Baker & Botts, LLP -r Q 


V^rlr NY 10119-09* tiffin ^ 




ETATS-UNIS D'AMERIQllE UiML 






Date of mailing (day/month/year) 

10 January 2000 (10.01.00) 


Applicant's or agent's file reference 
32095-PCT 


IMPORTANT NOTIFICATION 


International application No. 
PCT/US99/26125 


International filing date (day/month/year) 

05 November 1999 (05.11.99) 


International publication date (day/month/year) 

Not yet published 


Priority date (day/month/year) 

06 November 1998 (06.11.98) 


Applicant 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OFNEW YORK et al 





2. 



The applicant is hereby notified of the date of receipt (except where the letters IMR appear in tne rigm-nano coiumnj oy me 
International Bureau of the priority document(s) relating to the earlier application(s) indicated below. Unless otherwise 
indicated by an asterisk appearing next to a date of receipt or by the letters "NR", in the right-hand column, the priority 
document concerned was submitted or transmitted to the International Bureau in compliance with Rule 17.1(a) or (b). 

This updates and replaces any previously issued notification concerning submission or transmittal of priority documents. 

An asterisk^*) appearing next to a date of receipt in the right-hand column, denotes a priority document submitted 
or transmitted to the International Bureau but not in compliance with Rule 17.1 (a) or (b). In such a case, the attention 
of the applicant is directed to Rule 17.1(c) which provides that no designated Office may disregard the priority claim 
concerned before giving the applicant an opportunity, upon entry into the national phase, to furnish the priority document 
within a time limit which is reasonable under the circumstances. 

The letters "NR" appearing in the right-hand column denote a priority document which was not received by the International 

Bureau or which the applicant did not request the receiving Office to prepare and transmit to the International Bureau, 

as provided by Rule 17.1(a) or (b), respectively. In such a case, the attention of the applicant is directed to Rule 17.1(c) which 

provides that no designated Office may disregard the priority claim concerned before giving the applicant an opportunity, 

upon entry into the national phase, to furnish the priority document within a time limit which is reasonable under the 

circumstances. 



Priority date Priority application No. Country or regional Office Date of receipt 

or PCT receiving Office of priority document 

06 Nove 1998 (06.11.98) 60/107,463 US 28 Dece 1999 (28.12.99) 





Authorized officer 


The International Bureau of WIPO 




34, chemin des Colombettes 


Marc Salzman /CLjiX / 


1211 Geneva 20, Switzerland 




Facsimile No. (41-22) 740.14.35 


Telephone No. (41 -22) 338.83.38 V— ^/ 



Form PCT/IB/304 (July 1998) 003043706 



CM 
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ENT COOPERATION TREAl 



From the INTERNATIONAL SEARCHING AUTHORITY 



To: HENRY TANG 

BAKER & BOTTS, LLP 
30 ROCKEFELLER PLAZA 
NEW YORK NY 10112-0228 


PCT 

NOTIFICATION OF TRANSMITTAL OF 
THE INTERNATIONAL SEARCH REPORT 
OR THE DECLARATION 

(PCT Rule 44.1) 


S/»/XS g 0 3 APR 2000 


Applicant' 8 or agent's file reference 
32095-PCT 


FOR FURTHER ACTION See paragraphs 1 and 4 below 


International application No. 
PCT/US99/26125 


International filing date 
(day/ month fjcar) 

05 NOVEMBER 1999 


Applicant 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITI OF NEW YORK 



The applicant is hereby notified that the international search report has been established and is transmitted herewith. 
Filing of amendments and statement under Article 19: 

The applicant is entitled, if he so wishes, to amend the claims of the international application (see Rule 46): 

When? The time limit for filing such amendments is normally 2 months from the date of transmittal of the 
international search report; however, for more details, see the notes on the accompanying sheet. 

Where? Directly to the International Bureau of WIPO 
34, chemin des Colombettes 
1211 Geneva 20, Switzerland 
Facsimile No. : (41-22) 740. 14.35 

For more detailed instructions, see the notes on the accompanying sheet. 



2. 1 | The applicant is hereby notified that no international search report will be established and that the declaration under 
■ ■ Article 17(2)(a) to that effect is transmitted herewith. 



3. | \ With regard to the protest against payment of (an) additional fee(s) under Rule 40.2, the applicant is notified that: 

□ the protest together with the decision thereon has been transmitted to the International Bureau together with the 
applicant's request to forward the texts of ooth the protest and the decision thereon to the designated Offices. 

| | no decision has been made yet on the protest; the applicant will be notified as soon as a decision is made. 



4. Further action(s): The applicant is reminded of the following: 

Shortly alter 18 months from the priority date, the international application will be published by the International Bureau. 
If the applicant wishes to avoid or postpone publication, a notice of withdrawal of the international application, or of the 
priority claim, must reach the International Bureau as provided in rules 90 bis 1 and 90 bis 3, respectively, before the 
completion of the technical preparations for international publication. 

Within 19 months from the priority date, a demand for international preliminary examination must be fded if the applicant 
wishes to postpone the entry into the national phase until 30 months from the priority date (in some Offices even later). 

Within 20 months from the priority date, the applicant must perform the prescribed acts for entry into the national phase 
before all designated Offices which have not been elected in the demand or in a later election within 19 months from the 
priority date or could not be elected because they are not bound by Chapter II. 



\ \ \ / 


Name and mailing address of the ISA/US 

Commissioner of Patents and Trademarks 
Box PCT 

Washington, D.C 20231 
Facsimile No. (703) 305-3230 


Auth^^f^e^^/ 
troomas G. Black 

Telephone No. (703) 305-9707 



Form PCT/ISA/220 (July 1998)* (See notes on accompanying sheet) 



THIS PAGE BLANK («s"0) 



ENT COOPERATION TREA1 

PCT 

INTERNATIONAL SEARCH REPORT 

(PCT Article 18 and Rules 43 and 44) 



Applicant's or agent's file reference 
32095-PCT 


FOR FURTHER see Notification of Transmittal of International Search Report 
ACTION (Form PCT/ISA/220) as well as, where applicable, item 5 below. 


International application No. 
PCT/US99/26125 


International filing date (day /month/year) 
05 NOVEMBER 1999 


(Earliest) Priority Date (day/moruh/year) 
06 NOVEMBER 1998 


Applicant 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITI OF NEW YORK 



This international search report has been prepared by this International Searching Authority and is transmitted to the applicant 
according to Article 18. A copy is being transmitted to the International Bureau. 


This international search report consists of a total of / sheets. 




| X| It is also accompanied by a 


copy of each prior art document cited in this report. 


1. 


| | Certain claims were found unsearchable (See Box I). 


2. 


[ | Unity of invention is lacking (See Box II). 


3. 


1 I The international application contains disclosure of a nucleotide and/or amino acid sequence listing and the 
1—1 international search was carried out on the basis of the sequence listing 




□ 


filed with the international application. 




□ 


furnished by the applicant separately from the international application, 




□ 


| | but not accompanied by a statement to the effect that it did not include matter 
going beyond the disclosure in the international application as filed. 




transcribed by this Authority. 


4. 


With regard to the title, Qc] 


the text is approved as submitted by the applicant. 




□ 


the text has been established by this Authority to read as follows: 


5. 


With regard to the abstract, 






□ 


the text is approved as submitted by the applicant. 






the text has been established, according to Rule 38.2(b), by this Authority as it appears in 
Box III. The applicant may, within one month from the date of mailing of this international 
search report, submit comments to this Authority. 


6. 


The figure of the drawings to be published with the abstract is: 




Figure No. 1 j j 


as suggested by the applicant. I i VT „ . 

| | None of the figures. 

because the applicant failed to suggest a figure. 




□ 


because this figure better characterizes the invention. 
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InT^^onal application No. 
PCT/US99/26125 



Box III TEXT OF THE ABSTRACT (Continuation of item 5 of the first sheet) 



Systems and methods for generating standard description records from multimedia 
information are provided. The system includes at least one multimedia information 
input interface (180) receiving multimedia information, a computer processor, and 
a data storage system(150), operatively coupled to said processor, for storing said 
at least one description record. The processor performs object extraction processing 
to generate multimedia object descriptions (200, 201, 205) from the multimedia 
information , and object hierarchy processing (410, 420) to generate multimedia 
object hierarchy descriptions, to generate at least one description record including 
the multimedia object descriptions (200, 201, 205) and multimedia object hierarchy 
descriptions for content enbedded within the multimedia information. 
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INTERNATION, 



ARCH REPORT 



Inj^^Honal application No. 
PCmJS99/26125 



A. CLASSIFICATION OF SUBJECT MATTER 
IPC(6) :G06F/1730 

US CL : 707/10, 3, 4, 5, 104; 386/69; 395/806 
According to International Patent Classification (IPC) or to both national classification and IPC 



r I ELDS SEARCHED 



Minimum documentation searched (classification system followed by classification symbols) 
U.S. : 707/10, 3, 4, 5, 104; 386/69; 395/806 



Documentation searched other than minL^um documentation to the extent that such documents are included in the fields searched 



Electronic data base consulted during the international search (name of data base and, where practicable, search terms used) 
WEST, EAST 



DOCUMENTS CONSIDERED TO BE RELEVANT 



Category* 



Citation of document, with indication, where appropriate, of the relevant passages 



Relevant to claim No. 



Y 
Y 

Y 
Y 



US 5,613,032 A (CRUZ et al.) 18 March 1997, col. 3, line 30-col. 
4, line 44). 

US 5,623,690 A (PALMER et al) 22 April 1997, col. 1, lines 52-66. 

US 5,630,121 A (BRADEN-HARDER et al.) 13 May 1997, col. 2, 
line 54-col. 3, line 28. 

US 5,696,964 A (COX et al) 09 December 1997, col. 1, lines 8-11. 

US 5,701,510 A (JOHNSON et al) 23 December 1997, col. 1, line 
51-col. 2, line 17. 

US 5,713,021 A (KONDO et al) 27 January 1998, col. 1, line 55- 
col. 2, line 14. 



1-43 

1-43 
1-43 

1 

1-43 
1-43 



| x| Further documents are listed in the continuation of Box C. [ | See patent family annex. 



Special categories of cited document*: 

document defining the general state of the art which is not considered 
to be of particular relevance 

earlier document published on or after the international filing date 

document which may throw doubts on priority ciaim(s) or which is 
cited to establish the publication date of another citation or other 
special reason (as specified) 



document referring to an oral disclosure, use, exhibition or other 



document published prior to the international filing date but later than 
the priority date claimed 



later document published after the international filing date or priority 
date and not in conflict with the application but cited to understand 
the principle or theory underlying the invention 

document of particular relevance; the claimed invention cannot be 
considered novel or cannot be considered to involve an inventive step 
when the document is taken alone 

document of particular relevance; the claimed invention cannot be 
considered to involve an inventive step when the document is 
combined with one or more other such documents, such combination 
being obvious to a person skilled in the art 

document member of the same patent family 



Date of the actual completion of the international search 
10 FEBRUARY 2000 



Date of mailing of the international search report 




0 3 APR 2000 



Name and mailing address of the ISA/US 
Commissioner of Patents and Trademarks 
Box PCT 

Washington, D.C. 20231 
Facsimile No. (703) 305-3230 



iliomas G. Black 
Telephone No. (703) 305-9707 



Form PCT/ISA/210 (second sheet)(July 1992)* 
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InternHonai application No. 
PCT/US99/26125 



C (Continuation). DOCUMENTS CONSIDERED TO BE RELEVANT 



Category* 



Citation of document, with indication, where appropriate, of the relevant passages 



Relevant to claim No. 



Y 
Y 



US 5,758,076 A (WU et al) 26 May 1998, col. 1, lines 6-11. 

US 5,805,804 A (LAURSEN et al) 08 September 1998, col. 2, 
lines 14-38. 

US 5,822,524 A (CHEN et al) 13 October 1998, col. 3, line40-col. 
4, line 44. 

US 5,642,477 A (DE CARMO et al) 24 june 1997, col. 1, lines 10- 
14. 

US 5,713,021 A (KONDO et al) 27 January 1998, col. 1, line 56- 
col. 2, line 14. 



1 

1-43 
1-43 
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REQUEST 



The undersigned requests that the present 

international application be processed 
according to the Patent Cooperation Treaty. 



ceiving Office use only 



International Application No. 



International Filing Date 09/8^^89^ 



Name of receiving Office and "PCT International Appl i cation" 





Applicant's or agent's file reference 32095-PCT 
(if desired) (12 characters maximum) 


Box No. I TITLE OF INVENTION 

SYSTEMS AND METHODS FOR INTEROPERABLE MULTIMEDIA CONTENT DESCRIPTIONS 


Box No. II APPLICANT 


Name and address: (Family name followed by given name; for a legal entity, full official 
The address must include postal code and name of country. The country of the address indicated in this 
Box is the applicant's State (that is, country) of residence if no State of residence is indicated below.) 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK 
Broadway & 11 6th Street 
New York, NY 10027 
US 


| 1 This person is also inventor. 


Telephone No. 


Facsimile No. 


Teleprinter No. 


State (that is, country) of nationality: 
US 


State (that is, country) of residence: 
US 


This person is applicant I I all designated FCT aI1 designated States except 1 1 the United States 1 1 the States indicated in 
for the purposes of: L_ J States Ic^J the United States of America 1 1 of America only 1 1 the Supplemental Box 


Box No. Ill FURTHER APPLICANT(S) AND/OR (FURTHER) INVENTOR(S) 


Name and address: (Family name followed by given name; for a legal entity, full official designation. 
The address must include postal code and name of country. The country of the address indicated in this 
Box is the applicants State (that is, country) of residence if no State of residence is indicated below.) 

PAEK, SEUNGYUP 
530 Riverside Drive, Apt. 6J 
New York, NY 10027 
US 


This person is: 

| | applicant only 

1X1 applicant and inventor 

I | inventor only (If this check-box 
is marked, do not fill in below.) 


State (that is, country) of nationality: 
KR 


State (that is, country) of residence: 
US 


This person is applicant I I all designated I I all designated States except FCT tne United States I I the States indicated in 
for the purposes of: I— J States L_l the United States of America l£2u of America only I I the Supplemental Box 


^3 Further applicants and/or (further) inventors are indicated on a continuation sheet. 


Box No. IV AGENT OR COMMON REPRESENTATIVE; OR ADDRESS FOR CORRESPONDENCE 


The person identified below is hereby/has been appointed to act on behalf I — | 

of the applicants) before the competent International Authorities as: l&l a S ent I | common representative 


Name and address: (Family name followed by given name; for a legal entity, full official 
designation. The address must include postal code and name of country.) 

TANG, HENRY 

Baker & Botts, LLP r 

30 Rockefeller Plaza 

New York, NY 10112-0228 

US 


Telephone No. 
(212) 705-5000 


Facsimile No. 
(212) 705-5020 


Teleprinter No. 


| I Address for correspondence: Mark this check-box where no agent or common representative is/has been appointed and the 
I — I space above is used instead to indicate a special address to which correspondence should be sent. 



Form PCT/RO/I01 (first sheet) (July 1998; reprint July 1999) 



LegalSlar 1999, Form PCTREQ 



See Notes to the request form 
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Continuation of Box No. 



Sheet No. 



THER APPLICANTS AND/OR (FURTTOR) INVENTOR(S) 



If none of the following sub-boxes is used, this sheet is not to be included in the request 


Name and address: (Family name followed by given name; for a legal entity, full official designation. 
The address must include postal code and name of country. The country of the address indicated in this 
Box is the applicants State (that is, country) of residence if no State of residence is indicated below.) 

BENITEZ, ANA 

400 West 1 19th Street, Apt. 9F 

New York, NY 10027 

US 


This person is: 

| | applicant only 

|X| applicant and inventor 

| | inventor onlv (If this check-box 
is marked, do not fill in below.) 


State (that is, country) of nationality: 
ES 


State (that is, country) of residence: 
US 


This person is applicant I I all designated I I all designated States except Kj7j the United States I I the States indicated in 
for the purposes of: I I States I I the United States of America of America only I I the Supplemental Box 


Name and address: (Family name followed by given name; for a legal entity, full official designation. 
The address must include postal code and name of country. The country of the address indicated in this 
Box is the applicant's State (that is, country) of residence if no State of residence is indicated below.) 

CHANG, SHIH-FU 

560 Riverside Drive, Apt. 18K 

New York, NY 10027 

US 


This person is: 

| | applicant only 

[XI applicant and inventor 

| | inventor onlv (If this check-box 
is marked, do not fill in below.) 


State (that is, country) of nationality: 
TW 


State (that is, country) of residence: 

us 


This person is applicant I I all designated I I all designated States except K7I the United States I | the States indicated in 
for the purposes of: I — I States I — I the United States of America Ic^l of America only I I the Supplemental Box 


Name and address: (Family name followed by given name; for a legal entity, full official designation. 
The address must include postal code and name of country. The country of the address indicated in this 
Box is the applicant's State (that is, country) of residence if no State of residence is indicated below.) 


This person is: 

| | applicant only 

| | applicant and inventor 

| | inventor onlv (If this check-box 
is marked, do not fill in below.) 


State (that is, country) of nationality: 


State (that is, country) of residence: 


This person is applicant I I all designated I I all designated States except I I the United States I | the States indicated in 
for the purposes of: I — I States L_l the United States of America I I of America only I I the Supplemental Box 


Name and address: (Family name followed by given name; for a legal entity, full official designation. 
The address must include postal code and name of country. The country of the address indicated in this 
Box is the applicants State (that is, country) of residence if no State of residence is indicated below.) 


This person is: 

| | applicant only 

| | applicant and inventor 

| | inventor onlv (If this check-box 
is marked, do not fill in below.) 


State (that is, country) of nationality: 


State (that is, country) of residence: 


This person is applicant I I all designated I I all designated States except I | the United States I I the States indicated in 
for the purposes of: I — I States I — I the United States of America I I of America only l_J the Supplemental Box 


| | Further applicants and/or (further) inventors are indicated on another continuation sheet. 
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Box No.V DESIGNATIQ^ 



STATES 



The following designations are hereby made under Rule 4.9(a) (mark the applicable check-boxes; at least one must be marked): 
Regional Patent 

J3 AP ARIPO Patent: GH Ghana, GM Gambia, KE Kenya, LS Lesotho, MW Malawi, SD Sudan, SL Sierra Leone 
SZ Swaziland, UG Uganda, ZW Zimbabwe, and any other State which is a Contracting State of the Harare Protocol anc 
of the PCT 

g] EA Eurasian Patent: AM Armenia, AZ Azerbaijan, BY Belarus, KG Kyrgyzstan, KZ Kazakhstan, MD Republic of 
Moldova, RU Russian Federation, TJ Tajikistan, TM Turkmenistan, and any other State which is a Contracting State of 

g| EP European Patent: AT Austria, BE Belgium, CH and LI Switzerland and Liechtenstein, CY Cyprus, DE Germany, DK 
Denmark, ES Spain, FI Finland, TR France, GB United Kingdom, GR Greece, IE Ireland, IT Italy, LU 
Luxembourg, MC Monaco, NL Netherlands, PT Portugal, SE Sweden, and any other State which is a Contracting State 
of the European Patent Convention and of the PCT 

Jg| OA OAPI Patent: BF Burkina Faso, BJ Benin, CF Central African Republic, CG Congo, CI Cote d'lvoire, CM Cameroon 
GA Gabon, GN Guinea, GW Guinea-Bissau, ML Mali, MR Mauritania, NE Niger, §N Senegal, TD Chad TG Togo anc 
any other State which is a member State of OAPI and a Contracting State of the PCT (if other kind of protection or 



treatment desired, specify on dotted line) 



National Patent (if other kind of protection or treatment desired, specify on dotted line): 

63 AE United Arab Emirates 13 LR Liberia 

Albania 13 

Armenia [3 



ia 

El 
81 

ia 
ia 
ia 

ia 

ia 
ia 
ia 
ia 

81 

ia 

13 
81 

ia 
ia 
ia 

ia 

El 

ia 

ia 
ia 
ia 
8 

S3 
E 

81 
13 
81 

ia 



AL 
AM 
AT 
AU 
AZ 
BA 
BB 
BG 
BR 
BY 
CA 



Austria 

Australia 

Azerbaijan 

Bosnia and Herzegovina 

Barbados 

Bulgaria 

Brazil 

Belarus 

Canada 

CH and LI Switzerland and Liechtenstein 

CN China 

CU Cuba 

CZ Czech Republic 

DE Germany 

DK Denmark 

EE Estonia 

ES Spain 

FI Finland 

GB United Kingdom 
GD Grenada 

GE Georgia 

GH Ghana 

GM Gambia 

HR Croatia 

HU Hungary 
ID Indonesia 

IL Israel 

IN India 

IS Iceland 

JP Japan 

KE Kenya 

KG Kyrgyzstan 

KP Democratic People's Republic of Korea 



LS Lesotho 

LT Lithuania 
LU Luxembourg 
LV Latvia 

MD Republic of Moldova 

MG Madagascar 

MK The former Yugoslav Republic of Macedonia 



KR 
KZ 
LC 
LK 



Republic of Korea 

Kazakhstan 

Saint Lucia 
Sri Lanka 



81 
81 
81 
S3 

8! 
K> 
IS 
81 
8i 
81 
8! 
81 
81 
8i 
8! 

SI 
81 
81 
81 
81 
81 
81 
81 
81 
81 

81 
81 
81 
81 
81 

Check-boxes reserved for designating States which have become 
party to the PCT after issuance of this sheet: 

81 CR Costa Rica 13 DM. Dominica 

81 TZ Tanzania fiT MA .Morocco 



MN Mongolia 

MW Malawi 

MX Mexico 

NO Norway 

New Zealand 

Poland 

Portugal 

Romania 

Russian Federation 
Sudan 
Sweden 
Singapore 

Slovenia 

Slovakia 

Sierra Leone 

Tajikistan 

Turkmenistan 

Turkey 

Trinidad and Tobago 

Ukraine 

Uganda 

United States of America 



NZ 
PL 
PT 
RO 
RU 
SD 
SE 
SG 
SI 
SK 
SL 
TJ 
TM 
TR 
TT 
UA 
UG 
US 



UZ 
VN 
YU 
ZA 
ZW 



Uzbekistan 
Viet Nam 
Yugoslavia 
South Africa 
Zimbabwe 



Precautionary Designation Statement: In addition to the designations made above, the applicant also makes under Rule 4.9(b) all 
other designations which would be permitted under the PCT except any designation(s) indicated in the Supplemental Box as being 
excluded from the scope of this statement. The applicant declares that those additional designations are subject to confirmation and 
that any designation which is not confirmed before the expiration of 15 months from the priority date is to be regarded as withdrawn 
by the applicant at the expiration of that time limit. (Confirmation of a designation consists of the filing of a notice specifying that designation 
and the payment of the designation and confirmation fees. Confirmation must reach the receiving Office within the 15-month time limit.) 
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Sheet No 



(day/month/year) 


of earlier application 


national application: 
country 


regional application:* 
regional Office 


international application: 
receiving Office 


item (1) 

06 November 1 998 
(06.11.98) 


60/107,463 


US 






item (1\ 










item (3) 











Box No. VI PRIORITY,; 



D r urther priority ^^Bareindicated in the Supplemental Box. 



|N^| The receiving Office is requested to prepare and transmit to the International Bureau a certified copy 
It^l of the earlier application(s) (only if the earlier application was filed with the Office which for the 

purposes of the present international application is the receiving Office) identified above as item(s): (1) 

♦ Where the earlier application is an ARIPO application, it is mandatory to indicate in the Supplemental Box at least one country party to the Paris Convention for the 
Protection oj Industrial Property for which that earlier application was filed (Rule 4.IO(b)(ii)). See Supplemental Box. 



Box No. VH INTERNATIONAL SEARCHING AUTHORITY 



Choice of International Searching Authority (ISA) 

(if two or more International Searching Authorities are 
competent to carry out the international search, indicate the 
Authority chosen; the two~letter code may be used): 

ISA/ us 



Request to use results of earlier search; reference to that search (if an earlier 
search has been carried out by or requested from the International Searching Authority): 

Date (day/month/year) Number Country (or regional Office) 



Box No. VIII CHECK LIST: LANGUAGE OF FILING 



This international application contains 
the following number of sheets: 



request 

description (excluding) 
sequence listing part) 

claims 

abstract 

drawings 

sequence listing part 
of description 

Total number of sheets 



76 



This international application is accompanied by the item(s) marked below: 





i.H 


4 


2.0 




3.D 


52 


4.D 


11 


5.D 


1 


6.D 


8 


7.D 








9.E3 



Figure of the drawings which 
should accompany the abstract: 



Language of filing of the 

international application: 



English 



Box No. IX SIGNATURE OF APPLICANT OR AGENT 

ofoiowfrom il^n^he^quesj^ °f the P erson si S nin S and the capacity in which the person signs (if such capacity is not 




1 . Date of actual receipt of the purported 
international application: 


2. Drawings: 


3. Corrected date of actual receipt due to later but 
timely received papers or drawings completing the 
purported international application: 


| | received: 


4. Date of timely receipt of the required 
corrections under PCT Article 1 1(2): 


| | not received: 


5. International Searching Authority TC A / 
(if two or more are competent) : 1 ISA/ 


6. 1 ~1 Transmittal of search copy delayed 
1 — 1 until search fee is paid 




Date of receipt of the record copy 
by the International Bureau: 
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PCT 



FEE CALCULATION SHEET 
Annex to the Request 



Applicant's or agent's 
file reference 



32095-PCT 



For receiving Office use only 



International application No. 



Date stamp of the receiving Office 



Applicant 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK 



CALCULATION OF PRESCRIBED FEES 
1. TRANSMITTAL FEE 



240.00 



US 



450.00 



2. SEARCH FEE 

Internationa] search to be carried out by 

(If two or more International Searching Authorities are competent in relation to the international 
application indicate the name of the Authority which is chosen to carry out the international search') 

3. INTERNATIONAL FEE 
Basic Fee 

The international application contains 76 sheets, 
first 30 sheets 

x $10.00 



455.00 



46 x 

remaining sheets additional amount 



460, 



.00 ft>2 



Add amounts entered at b } and b 2 uid enter total at B 
Designation Fees 

The international application contains designations. 



915.00 



10 



105.00 



number of designation fees amount of designation fee 
payable (maximum 10) 



Add amounts entered at B and D and enter total at I 



1, 050.00 [^] 



1,965.00 I 



(Applicants from certain States are entitled to a reduction of 75% of the 
international fee. Where the applicant is (or all applicants are) so entitled, 

4. FEE FOR PRIORITY DOCUMENT (if applicable) : 

5. TOTAL FEES PAYABLE 
Add amounts entered at T, S, I and P, and enter total in the TOTAL box 



15.00 



2,900.00 



TOTAL 



□ 



The designation fees are not paid at this time. 



MODE OF PAYMENT 

□ authorization to charge 
deposit account (see below) 

|XI cheque 

postal money order 



| | bank draft 
| | cash 

□ 

revenue stamps 



| | coupons 

| | other (specify): 



DEPOSIT ACCOUNT AUTHORIZATION (this mode of payment may not be available at ail receiving Offices) 



The RO/ us 



account. 



02-4377 



| | is hereby authorized to charge the total fees indicated above to my deposit 

(this check-box may be marked only if the conditions for deposit accounts of the receiving Office so permit) is 
hereby authorized to charge any deficiency or credit any overpayment in the total fees indicated above to my 
deposit account. 

□ is hereby authorized to charge the fee for preparation and trangfnjttal of the priority document to the International 
Bureau of WIPO to my deposit account. 

5 November 1999 



Deposit Account Number 



Date (day/month/year) 




Signature 
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See Notes to the fee calculation sheet 
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From the RECEIVING OFFICE 



TENT COOPERATION TRE/ 



To: 

HENRY TANG 

BAKER & BOTTS, LLP 

30 ROCKEFELLER PLAZA 

uru vnDV uv 1A119 

flCW TUKK NT U££0 


PCT 

NOTIFICATION OF THE INTERNATIONAL 
APPLICATION NUMBER AND OF THE 
INTERNATIONAL FILING DATfc, 

(PCT Rule 20.5(c)) 


Date of mailing , 

{daylmonthfyear) Q Q Q£Q flQQ 


Applicant's or agent's file reference 
32095-PCT 


IMPORTANT NOTIFICATION 


International application No. International filing date (daylmonthfyear) 

PCT/US99/26125 05 NOV 99 


Priority date {daylmonthfyear) 
06 NOV 98 


Applicant THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CI 
NEW YORK 


TY OF 


Title of the invention SYSTEMS AND METHODS FOR INTEROPERABLE MULTIMEDIA 
CONTENT DESCRIPTIONS 



1. The applicant is hereby notified that the international application has been accorded the international application number and 
the international riling date indicated above. 



2. The applicants further notified that the record copy of the international application: 

was transmitted to the International Bureau on 0 9 DEC Iflflfl 



; applicant^ 



| I has not yet been transmitted to the International Bureau for the reason indicated below and a copy of this 
■ — ■ notification has been sent to the International Bureau •: 

[ | because the necessary national security clearance has not yet been obtained. 
| | because {reason to be specified): 



* The International Bureau monitors the transmittal of the record copy by the receiving Office and will notify the applicant 
(with Form PCT/1B/30 1 ) of its receipt. Should the record copy not have been received by the expiration of 14 months from 
the priority date, the International Bureau will notify the applicant (Rule 22.1(c)). 



FOREIGN TRANSMITTAL LICENSE INFORMATION 

□ 



Completed 



Additional license for foreign transmittal not required. This subject matter is covered by a license already 
granted on the equivalent U.S. national application. Refer to that license for information concerning its scope. 

□ License for foreign transmittal not required. 37 CFR 5.1 1(e)(1) or 37 CFR 5.1 1(e)(2). However, a license may 
ybe required for additional subject matter. See 37 CFR 5.15(b). 

\TY Foreign transmittal license granted. 35 U.S.C. 184; 37 CFR 5.11 on 
1 — 1 / (date) 



[17] 37 CFR 5.15(a) 



| | 37 CFR 5.15(b) 





Name and mailing address of the receiving Office 
Assistant Commissioner for Patents 
Box PCT 

Washington, D.C. 20231 Attn: RO/US 
Facsimile No. 


Authorized officer .. , , JJHi 

Felicia Lawrenc^^/[ 
^CT Operations -wuiean 1 
703)305-3675 (703) 305-3230 (RAX) 

Telephone No. 



Form PCT/RO/105 (July 1992) 
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PCT 



TENT COOPERATION TR^^Y 

From the INTERNATIONAL BUREAU 



PCT/US99/26125 

LOT 




NOTIFICATION OF RECEIPT OF 
RECORD COPY 

(PCT Rule 24.2(a)) 



To: 



TANG, Henry 
Baker & Botts, LLP 
30 Rockefeller Plaza 
New York, NY 1 01 1 2-0228 

ETATS-UNIS D 1 AMERIQ^ KER BOTTS L.L.R 



00 JAN 1 1 AM10:U2 



Date of mailing (day/month/year) 

20 December 1999 (20.12.99) 



IMPORTANT NOTIFICATION' 




Applicant's or agent's file reference 
32095-PCT 



International application No. 
PCT/US99/26125 



The applicant is hereby notified that the International Bureau has received the record copy of the international application as 
detailed below. 

Name(s) of the applicant(s) and State(s) for which they are applicants: 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OFNEW YORK (for all designated 
States except US) 

PAEK, Seungyup et al (for US) 
International filing date 05 November 1999 (05.1 1 .99) 

Priority date(s) claimed 06 November 1998 (06.11.98) 

Date of receipt of the record copy 

by the International Bureau 13 December 1999 (13.12.99) 

List of designated Offices : 

AP :GH,GM,KE,LS,MW,SD,SL,SZ/re,UG,ZW 
EA :AM,AZ,BY,KG,KZ,MD,RU,TJ,TM 

EP rAT^^CH.CY^D^DK^S^lFR^B^RJEJT^LU^C^UPT^SE 
OA iBF^BJ^CF^CCCI^M^A^N^GW^UMR^E^SNTD.TG 

National ^E.AUAM^AU.AZ^A^BB.BG^R^Y^CH^N^CR^UCZ^D^D^DM^EE^ES^F^GB, 

GD,GE,GH,GM,HR,HU,IDJUN,ISJP,KE,KG,KP,KR^ 

MN,MW,MX,NO,NZ,PL,PT,RO,RU,SD,SE,SG,SI,SK,SL/rj^^ 

ZW 



COPY TO 



r 


The International Bureau of WIPO 
34, chemin des Colombettes 
1211 Geneva 20, Switzerland 

Facsimile No. (41-22)740.14.35 


Authorized officer: ^ 

J. Leitao 

Telephone No. (41-22)338.83.38 



Form PCT/IB/301 (July 1998) 



003029754 



Continuation of Form 
NOTIFICATION OF RECEIPT OF RECORD COPY 



PCT/US99/26125 



Date of mailing (day/month/year) 

20 December 1999 (20.12.99) 


IMPORTANT NOTIFICATION 


Applicant's or agent* s file reference 
32095-PCT 


International application No. 
PCT/US99/26125 



ATTENTION 

The applicant should carefully check the data appearing in this Notification. In case of any discrepancy between these data 
and the indications in the international application, the applicant should immediately inform the International Bureau. 

In addition, the applicant's attention is drawn to the information contained in the Annex, relating to: 
| X| time limits for entry into the national phase 
| | confirmation of precautionary designations 
| X| requirements regarding priority documents 
A copy of this Notification is being sent to the receiving Office and to the Internationa (Searching Authority. 
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International application No. 

PCT/US99/26125 



ANNE._4P FORM PCT/IB/301 





INFORMATION ON TIME LIMITS FOR ENTERING THE NATIONAL PHASE 



The applicant is reminded that the "national phase" must be entered before each of the designated Offices indicated in the 
Notification of Receipt of Record Copy (Form PCT/IB/301) by paying national fees and furnishing translations, as prescribed by 
the applicable national laws. 

The time limit for performing these procedural acts is 20 MONTHS from the priority date or, for those designated States 
which the applicant elects in a demand for international preliminary examination or in a later election, 30 MONTHS from the 
priority date, provided that the election is made before the expiration of 19 months from the priority date. Some designated (or 
elected) Offices have fixed time limits which expire even later than 20 or 30 months from the priority date. In other Offices an . 
extension of time or grace period, in some cases upon payment of an additional fee, is available. 

In addition to these procedural acts, the applicant may also have to comply with other special requirements applicable in 
certain Offices. It is the applicant's responsibility to ensure that the necessary steps to enter the national phase are taken in a 
timely fashion. Most designated Offices do not issue reminders to applicants in connection with the entry into the national 
phase. - 

For detailed information about the procedural acts to be performed to enter the national phase before each designated 
Office, the applicable time limits and possible extensions of time or grace periods, and any other requirements, see the relevant 
Chapters of Volume II of the PCT Applicant's Guide. Information about the requirements for filing a demand for international 
preliminary examination is set out in Chapter IX of Volume I of the PCT Applicant's Guide. 

GR and ES became bound by PCT Chapter II on 7 September 1996 and 6 September 1997, respectively, and may, therefore, 
be elected in a demand or a later election filed on or after 7 September 1996 and 6 September 1997, respectively, regardless of 
the filing date of the international application. (See second paragraph above.) 

Note that only an applicant who is a national or resident of a PCT Contracting State which is bound by Chapter II has 
the right to file a demand for international preliminary examination. 



This notification lists only specific designations made under Rule 4.9(a) in the request. It is important to check that these 
designations are correct Errors in designations can be corrected where precautionary designations have been made under 
Rule 4.9(b). The applicant is hereby reminded that any precautionary designations may be confirmed according to Rule 4.9(c) 
before the expiration of 15 months from the priority date. If it is not confirmed, it will automatically be regarded as withdrawn 
by the applicant. There will be no reminder and no invitation. Confirmation of a designation consists of the filing of a notice 
specifying the designated State concerned (with an indication of the kind of protection or treatment desired) and the payment 
of the designation and confirmation fees. Confirmation must reach the receiving Office within the 15-month time limit 



For applicants who have not yet complied with the requirements regarding priority documents, the following is recalled. 

Where the priority of an earlier national, regional or international application is claimed, the applicant must submit a copy 
of the said earlier application, certified by the authority with which it was filed ("the priority document") to the receiving Office 
(which will transmit it to the International Bureau) or directly to the International Bureau, before the expiration of 16 months from 
the priority date, provided that any such priority document may still be submitted to the International Bureau before that date of 
international publication of the international application, in which case that document will be considered to have been received 
by the International Bureau on the last day of the 16-month time limit (Rule 17.1(a)). 

Where the priority document is issued by the receiving Office, the applicant may, instead of submitting the priority 
document, request the receiving Office to prepare and transmit the priority document to the International Bureau. Such request 
must be made before the expiration of the 16-month time limit and may be subjected by the receiving Office to the payment 
of a fee (Rule 17.1(b)). 

If the priority document concerned is not submitted to the International Bureau or if the request to the receiving Office 
to prepare and transmit the priority document has not been made (and the corresponding fee, if any, paid) within the applicable 
time limit indicated under the preceding paragraphs, any designated State may disregard the priority claim, provided that no 
designated Office may disregard the priority claim concerned before giving the applicant an opportunity to furnish the priority 
document within a time limit which is reasonable under the circumstances. 

Where several priorities are claimed, the priority date to be considered for the purposes of computing the 16-month time 
limit is the filing date of the earliest application whose priority is claimed. 
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LATENT COOPERATION TRE 



From the 

INTERNATIONAL PRELIMINARY EXAMINING AUTHORITY 



To: HENRY TANG 

BAKER BOTTS LLP 

30 ROCKEFELLER PLAZA 

NEW YORK, NY 10112-0228 



NOTIFICATION OF T^^SMil^At 1 ©^ ^ 
INTERNATIONAL PRELIMINARY ; 
EXAMINATION-JlEP^^--^R 




Date of Mailing 
(day/ month /jear) 



07DECZ000 fitfcf 



Applicant's or agent's file .reference 
32095-PCT 


IMPORTANT NOTIFICATION 


International application No. 
PCT/US99/26125 


International filing date (day/ month /jeor) 
05 NOVEMBER 1999 


Priority Date (day/ month /year) 
06 NOVEMBER 1998 


Applicant 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITI OF NEW YORK 



1 . The applicant is hereby notified that this International Preliminary Examining Authority transmits herewith the 
international preliminary examination report and its annexes, if any, established on the international application. 

2. A copy of the report and its annexes, if any , is being transmitted to the International Bureau for communication 
to all the elected Offices. 

3. Where required by any of the elected Offices, the International Bureau will prepare an English translation of 
the report (but not of any annexes) and will transmit such translation to those Offices. 

4. REMINDER 

The applicant must enter the national phase before each elected Office by performing certain acts (filing 
translations and paying national fees) within 30 months from the priority date (or later in some Offices)(Article 
39(l))(see also the reminder sent by the International Bureau with Form PCT/IB/301). 

Where a translation of the international application must be furnished to an elected Office, that translation must 
contain a translation of any annexes to the international preliminary examination report. It is the applicant's 
responsibility to prepare and furnish such translation directly to each, elected Office concerned. 

For further details on the applicable time limits and requirements of the elected Offices, see Volume II of the 
PCT Applicant's Guide. 



OH DOLfSftf KM n 






Name and mailing address of the I PEA/ US 

Commissioner of Patents and Trademarks 
Box PCT 

Washington, D.C. 20231 
Facsimile No. (703) 305-3230 


Authorized officer 

Thomas G. Black * 

Telephone No. (703) §05-9707 





Form PCT/IPEA/416 (July 1992)* 
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"ATENT COOPERATION TREATY 

PCT 

INTERNATIONAL PRELIMINARY EXAMINATION REPORT 
(PCT Article 36 and Rule 70) 



Applicant's or agent's file reference 
32095-PCT 


FOR FURTHER ACTION Scc Notification of Transmittal of International 

Preliminary Examination Report (Form PCT/IPEA/416) 


International application No. 
PCT/US99/26125 


International filing date (day/ month /j ear) 
05 NOVEMBER 1999 


Priority date (day/ month /year) 
06 NOVEMBER 1998 


International Patent Classification (IPC) or national classification and IPC 
IPC(7): G06F 17/30 and US CI.: 707/10, 3, 4, 5, 104; 386/69; 395/806 


Applicant 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITI OF NEW YORK 



1. This international preliminary examination report has been prepared by this International Preliminary 
Examining Authority and is transmitted to the applicant according to Article 36. 

2. This REPORT consists of a total of ^3 sheets. 

I I This report is also accompanied by ANNEXES, i.e., sheets of the description, claims and/or drawings which have 
been amended and are the basis for this report and/or sheets containing rectifications made before this Authority, 
(see Rule 70.16 and Section 607 of the Administrative Instructions under the PCT). 



These annexes consist of a total of ■ 



sheets. 



3. This report contains indications relating to the following items: 
Basis of the report 
Priority 

Non-establishment of report with regard to novelty 
Lack of unity of invention 

Reasoned statement under Article 35(2) with regard t< 
citations and explanations supporting such statement 



I 


0 


II 


□ 


III 


□ 


IV 


□ 


V 




VI 


□ 


VII 


□ 


VIII 


□ 



Date of submission of the demand 
17 MAY 2000 


Date of completion of this report 
08 AUGUST 2000 


Name and mailing address of the I PEA/ US 

Commissioner of Patents and Trademarks 
Box PCT 

Washington, D.C. 20231 
Facsimile No. (703) 305-3230 


Authorized officer 

Thomas G. Black ^Wfia /}1*d6hu& 

Telephone No. (703) 305-9707 



Form PCT/IPEA/409 (cover sheet) (July 1998)* 
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INTERNATIONAL PRELIMINARY EXAMINATION REPORT 



International application No. 
PCT/US99/26125 



I. Basis of the report 



1. With regard to the elements of the international application:* 
| X | the international application as originally filed 
p^Tj the description: 

pages 

pages 



1-49 



NONE 



pages . 



NONE 



, as originally filed 

, filed with the demand 



, filed with the letter of 



| x| the claims: 

pages 50-60 

pages NONE 

pages NONE 

pages 



, as originally filed 

. , as amended (together with any statement) under Article 1 9 
, filed with the demand 



NONE 



filed with the letter of 



j x| the drawings: 

pages 1-6 

pages 

pages 



NONE 



, as originally filed 

filed with the demand 



NONE 



| x| the sequence listing part of the description: 

pages NONE 

pages 

pages 



, filed with the letter of . 



NONE 



, as originally filed 

, filed with the demand 



NONE 



, filed with the letter of . 



2. With regard to the language, all the elements marked above were available or furnished to this Authority in the language in which 
the international application was filed, unless otherwise indicated under this item. 

These elements were available or furnished to this Authority in the following language which is: 

I I the language of a translation furnished for the purposes of international search (under Rule 23.1(b)). 
I I the language of publication of the international application (under Rule 48.3(b)). 

1 j the language of the translation furnished for the purposes of international preliniinary examination (under Rules 55.2 and/ 
or 55.3). 

3. With regard to any nucleotide and/or amino acid sequence disclosed in the international application, the international 
preliminary examination was carried out on the basis of the sequence listing: 

□ contained in the international application in printed form. 

| | filed together with the international application in computer readable form. 

| | furnished subsequently to this Authority in written form. 

| | furnished subsequently to this Authority in computer readable form. 

I I Th e statement that the subsequently furnished written sequence listing does not go beyond the disclosure in the 

1 — 1 international application as filed has been furnished. 

I 1 Th e statement that the information recorded in computer readable form is identical to the writen sequence listing, has 

1 — 1 been furnished. 

4 1 x| The amendments have resulted in the cancellation of: 

the description, pages NONE 

the claims, Nos. 



NONE 

the drawings, sheets/fig NONE 



This report has been drawn as if (some of) the amendments had not been made, since they have been considered to go 
beyond the disclosure as filed, as indicated in the Supplemental Box (Rule 70.2(c)) ** 
* Replacement sheets^ which have been furnished to the receiving Office in response to an invitation under Article 14 are referred to 
in this report as "originally filed" and are not annexed to this report since they do not contain amendments (Rules 70.16 
and 70.17). r J ' 

*M/ry replacement sheet contai ning such amendments must be referred to under item 1 and annexed to this report. 
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INTERNATIONAL PRELIMINARY EXAMINATION REPORT 



International application No. 
PCT/US99/26125 



V. Reasoned statement under Article 35(2) with regard to novelty, inventive step or industrial applicability; 
citations and explanations supporting such statement 



1. statement 

Novelty (N) 



Claims 1-43 



Claims NONE 



YES 
NO 



Inventive Step (IS) 



Claims 1-43 



Claims NONE 



YES 
NO 



Industrial Applicability (IA) 



Claims 1-43 



Claims NONE 



YES 
NO 



citations and explanations (Rule 70.7) 

Claims 1-43 meet the criteria set out in PCT Article 33(2)-(4), because the prior art does not teach or fairly suggest a 
system for generating a description record from multimedia information wherein at least one description record including 
said multimedia object description and said multimedia object hierarchy description is generated for content embedded within 
said multimedia information. 



NEW CITATIONS 



NONE 
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PATENT COOPERATION T 



From the * 

INTERNATIONAL PREUMINA^ EXAMINING AUTHORITY 



Toe 



HENRY TANG 
BAKER & BOTTS, LLP 
30 ROCKEFELLER PLAZA 
NEW YORK NY 10112-0228 



per 



NOTIFICATION OFRECHDPT 
OF DEMAND BY COMPETENT INTERNATIOJ 

PRELIMINARY EXAMINING AUTHORITY ^^^^ 

(PCT Rule 593(e) and 6Ll(b), first saitenoe fa/jJ 
and Administrative Instructions, Section 601(a)) * j/^ls d ^- — 



Date of mailing 
(daylmomhfyear) 



22JUN2000 



Applicant's or agent's file reference 
32095-PCT 


IMPORTANT NOTIFICATION 


International application No. 

PCT/US99/26125 


International filing date ( day /month/year) 
05 NOV 99 


Priority date (day/month/year) 

06 NOV 98 


Applicant 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF 
NEW YORK 



1. The applicant is hereby notified that this International Preliminary Examining Authority considers the following date as the 
date of receipt of the demand for international preliminary examination of the international application: 



ff May JIM t /T AT&l 



2. That date of ffeceipt is: 

| y the actual date of receipt of the demand by this Authority (Rule 61.1(b)). 

| j the actual date of receipt of the demand on behalf of this Authority (Rule 593(e)). 



3. 



□ 



| I the date on which this Authority has, in response to the invitation to correct defects in the demand (Form 
PCT/IPEA/404), received the required corrections. 



ATTENTION: That date of receipt is AFTER the expiration of 19 months from the priority date. Consequently, the 
electton(s) made in the demand does (do) not have the effect of postponing the entry into the national phase until 
30 months from the priority date (or later in some Offices) (Article 39(1)). Therefore, the acts for entry into the 
national phase must be performed within 20 months from the priority date (or later in some Offices) (Article 22). 
For details, see the PCT Appiicani's Guide, Volume II. 



□ 



(If applicable) This notification confirms the information given by telephone, facsimile transmission or in person 
on: 



4. Only where paragraph 3 applies, a copy of this notification has been sent to the International Bucpp* &UO(flri RjA 



Name and mailing address of the IPEA/US 
Assistant Commissioner for Patents 
Box PCT 

Washington, D.C. 20231 Attn: IPEA/US 

Facsimile No. 



Authorized officer 



•Mntfnvv (\A 

Telephone No. (703)3053753 



Form PCT/IPEA/402 (July 1998) 



{ZQSH30S323QJ1B ' 
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PCT 



LATENT COOPERATION TR 




From the INTERNATIONAL BUREAU 



PCT/US9a£6125 



NOTIFICATION CONCERNING 
THE FILING OF AMENDMENTS OF THE CLAIMS 

(PCT Administrative Instructions, Section 417) 



Date of mailing 

(day/month/year) 1g Mgy 2QQ0 (16 05 . 0 0) 



To: 



TANG, Henry 
Baker & Botts, LLP 
30 Rockefeller Plaza 
New York, NY 101 12-0228 

ETATS-UNIS D'AMERIQU£ KER qqjjs L . L >. 



nnM&Y30 PH I* 59 



Applicant's or agent's file reference 
32095-PCT 



IMPORTANT NOTIFICAgQfc 



International application No. 

PCT/US99/26125 



International filing date i . \ rv\ 

(day/month/year) Q5 Novem beV l&W (Ub. I 1 .35) 



Applicant 



THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OFNEW YORK et al 



1. The applicant is hereby notified that amendments to the claims under Article 19 were received by the International Bureau on: 

12 May 2000(12.05.00) 

2. This date is within the time limit under Rule 46.1. 

Consequently, the international publication of the international application will contain the amended claims according 
to Rule 48.2(f), (h) and (i). 



3. The applicant is reminded that the international application (description, claims and drawings) may be amended during 
the international preliminary examination under Chapter II, according to Article 34, and in any case, before each of the 
designated Offices, according to Article 28 and Rule 52, or before each of the elected Offices, according to Article 41 
and Rule 78. 



The International Bureau of WIPO 
34, chemin des Colombettes 
121 1 Geneva 20, Switzerland 



Facsimile No.: (41-22) 740.14.35 



Authorised officer 



Dominique DELMj^: 
Telephone No.: (41-22)338.83.38 



Form PCT/IB/346 (September 1993) 



3287889 
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ENT COOPERATION TREA 



PCT 



INFORMATION CONCERNING ELECTED 
OFFICES NOTIFIED OF THEIR ELECTION 



(PCT Rule 61.3) 



Date of mailing (day/month/year) 
26 July 2000 (26.07.00) 



PCT/US99/26125 

3Z.CRS 

AKER BGTTS L.fe/ 

From the INTERNATIONAL f ^UflpAU 7 p M y>. 



# 




TANG, Henry 
Baker & Botts, LLP 

30 Rockefeller Plaza * lAfirrdr^ 
New York, NY 101 12-0228 \J^J 
ETATS-UNIS D'AMERIQUE 



Applicant's or agent's file reference 
32095-PCT 


IMPORTANT INFORMATION 


International application No. 
PCT/US99/26125 


International filing date (day/month/year) 
05 November 1999 (05.11.99) 


Priority date (day/month/year) 

06 November 1998 (06.11.98) 


Applicant 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OFNEW YORK et al 



1. The applicant is hereby informed that the International Bureau has, according to Article 31(7), notified each of the following 
Offices of its election: 

AP iGKGM^KE^S^MW^D^SL^ZT^UCZW 

EP iAT^^CH^DE^K^ES^IFR^GB^RJEJT^U^MCNL^T^SE 

National iAU^CBR^CA^CN^DEJLJP^KP^KR^N^O^^PL^CRU^E^K^US 

2. The following Offices have waived the requirement for the notification of their election; the notification will be sent to them 
by the International Bureau only upon their request: 

EA :AM,AZ,BY,KG,KZ,MD,RU,TJ,TM 

OA iBF^BJ^F^CCCLCM^GA^N^W^L^R^E^N^TDJG 

National :AE,AL,AM,AT # AZ / BA,BB f BY / CH / CR / CU,DK / DM / EE / ES / FI / GB,GD / GE,GH,GM # 
HR^UJDJNJS^KE^G^Z^C^K^R^S^LT^aLV^A^MaMG^MK^MW^X^T^SD^G, 
SI^LTJ^TM^TR^TT^UA^CUZ^VN^U^ZW 

3. The applicant is reminded that he must enter the "national phase" before the expiration of 30 months from the priority date 
before each of the Offices listed above. This must be done by paying the national fee(s) and furnishing , if prescribed, a 
translation of the international application (Article 39(1 )(a)), as well as, where applicable, by furnishing a translation of any 
annexes of the international preliminary examination report (Article 36(3)(b) and Rule 74.1). 

Some offices have fixed time limits expiring later than the above-mentioned time limit. For detailed information about the 
applicable time limits and the acts to be performed upon entry into the national phase before a particular Office, see Volume li 
of the PCT Applicant's Guide. 

The entry into the European regional phase is postponed until 31 months from the priority date for all States designated for 
the purposes of obtaining a European patent. 



ON r 




7 -v- 




The International Bureau of WIPO 
34, chemin des Colombettes 
1211 Geneva 20, Switzerland 



Facsimile No. (41-22) 740.14.35 



Authorized officer: 

Jean-Marie McAdams 

Telephone No. (41 -22) 338.83.38 



Form PCT/!3/332 (September 1997) 



3429621 



» # 
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PCT 



NOTICE INFORMING THE APPLICANT OF THE 
COMMUNICATION OF THE INTERNATIONAL 
APPLICATION TO THE DESIGNATED OFFICES 

(PCT Rule 47.1(c), first sentence) 



From the INTERNATIONAL BUREAU 



Date of mailing (day/month/year) 

18 May 2000(18.05.00) 



WO 00/28440 
PCT/US99/26125 



To: 



TANG, Henry 
Baker & Botts, LLP 
30 Rockefeller Plaza 
New York, NY 10112-0228 
ETATS-UNIS D'AMERIQUE 



BAKER BOTTS LLP. 
00 HAY 30 PH h 00 



Applicant's or agent's file reference 
32095-PCT 



IMPORTANT NOTICE 



International application No. 
PCT/US99/26125 



International filing date (day/month/year) 

05 November 1999 (05.11.99) 



06 November 




Priority date (jdayymonjjhyyearj 



Applicant 



THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OFNEW YORK et al 



1 . Notice is hereby given that the International Bureau has communicated, as provided in Article 20, the international application 
to the following designated Offices on the date indicated above as the date of mailing of this Notice: 

AU,CN,JP,KP,KR,MA,US 

In accordance with Rule 47.1(c), third sentence, those Offices will accept the present Notice as conclusive evidence that 
the communication of the international application has duly taken place on the date of mailing indicated above and no copy 
of the international application is required to be furnished by the applicant to the designated Office(s). 

2. The following designated Offices have waived the requirement for such a communication at this time: 

AE / AL,AM / AP / AT,AZ,BA,BB / BG,BR,BY,CA,CH,CR,CU / CZ,DE / DK / DM,EA / EE / EP / ES / FI / GB / GD / GE r 

GH^GM^R^HUJDJL^NJS.KE^CKZ^CL^LR^SX^LU^LV^MD.MCMK.MN^W^X^O^^OA, 

PL,PT,RQ,RU,SD,SE,SG,SI,SK,SL^ 

me communication will be made to those Offices only upon their request. Furthermore, those Offices do not require the 

applicant to furnish a copy of the international application (Rule 49.1 (a-bis)). 

3. Enclosed with this Notice is a^cjM^oHhe international application as published by the International Bureau on 
18 May 2000 (18.05.00) under No. WO 00/28440 



REMINDER REGARDING CHAPTER II (Article 31(2)(a) and Rule 54.2) 

If the applicant wishes to postpone entry into the national phase until 30 months (or later in some Offices) from the priority 
date, a demand for international preliminary examination must be filed with the competent International Preliminary 
Examining Authority before the expiration of 19 months from the priority date. 

It is the applicant's sole responsibility to monitor the 1 9-month time limit. 

Note that only an applicant who is a national or resident of a PCT Contracting State which is bound by Chapter II has the 
right to file a demand for international preliminary examination. 



REMINDER REGARDING ENTRY INTO THE NATIONAL PHASE (Article 22 or 39(1)) 

If the applicant wishes to proceed with the international application in the national phase, he must, within 20 months 
or 30 months, or later in some Offices, perform the acts referred to therein before each designated or elected Office. 

For further important information on the time limits and acts to be performed for entering the national phase, see the 
Annex to Form PCT/IB/301 (Notification of Receipt of Record Copy) and Volume II of the PCT Applicant's Guide. 



ON POCKET Erg 



The International Bureau of W1PO 
34, chemin des Colombettes 
1211 Geneva 20, Switzerland 



Facsimile No. (41-22) 740.14.35 



Authorized officer 

J. Zahra 

Telephone No. (41-22) 338.83.38 




Of 



Form PCT/IB/308 (July 1996) 



3277914 
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